{
  "id": 25243,
  "title": "F# script to create duplicate based features",
  "url": "/competitions/bosch-production-line-performance/discussion/25243",
  "author_name": "",
  "post_date": "2016-11-08T20:29:43.037Z",
  "votes": 4,
  "comment_count": 2,
  "views": 449,
  "content": "<p>Hi everyone,</p>\n\n<p>Feature engineering in F# is a lot of fun so I decided to create more &quot;magic&quot; features, this time based on duplicates.</p>\n\n<p>I define a duplicate in 3 ways:</p>\n\n<ol>\n<li><p>All numeric features are the same </p></li>\n<li><p>All date features are the same</p></li>\n<li><p>Both numeric and date features are the same</p></li>\n</ol>\n\n<p>I calculate hashcodes of the features and then simply group by the hashcode to find groups of identical rows across train and test.</p>\n\n<p>For each train and test record 3 features are then engineered:</p>\n\n<ol>\n<li><p>Number of duplicates in the duplicate group which contains the record. For most records this will be 1.</p></li>\n<li><p>Boolean flag which is True if this group has consecutive Ids (&quot;series&quot; - see my previous post)</p></li>\n<li><p>Number of faults in the group as int option - response of the record. It turns out if we know that a record has a duplicate that is faulty then the probability of it being faulty as well is about 38%:)</p></li>\n</ol>\n\n<p>The script is here:\n<a href=\"https://github.com/Statfactory/Kaggle_Bosch\">https://github.com/Statfactory/Kaggle_Bosch</a></p>\n\n<p>The script first creates a csv file, so you can use it in R or Python as well.</p>\n\n<p>For Xi modelling you will need to import it with Xi.ImportDataAsync</p>\n\n<p>Cheers</p>\n\n<p>Adam</p>",
  "messages": [
    {
      "id": "143444",
      "postDate": "11/08/2016 20:29:43",
      "content": "<p>Hi everyone,</p>\n\n<p>Feature engineering in F# is a lot of fun so I decided to create more &quot;magic&quot; features, this time based on duplicates.</p>\n\n<p>I define a duplicate in 3 ways:</p>\n\n<ol>\n<li><p>All numeric features are the same </p></li>\n<li><p>All date features are the same</p></li>\n<li><p>Both numeric and date features are the same</p></li>\n</ol>\n\n<p>I calculate hashcodes of the features and then simply group by the hashcode to find groups of identical rows across train and test.</p>\n\n<p>For each train and test record 3 features are then engineered:</p>\n\n<ol>\n<li><p>Number of duplicates in the duplicate group which contains the record. For most records this will be 1.</p></li>\n<li><p>Boolean flag which is True if this group has consecutive Ids (&quot;series&quot; - see my previous post)</p></li>\n<li><p>Number of faults in the group as int option - response of the record. It turns out if we know that a record has a duplicate that is faulty then the probability of it being faulty as well is about 38%:)</p></li>\n</ol>\n\n<p>The script is here:\n<a href=\"https://github.com/Statfactory/Kaggle_Bosch\">https://github.com/Statfactory/Kaggle_Bosch</a></p>\n\n<p>The script first creates a csv file, so you can use it in R or Python as well.</p>\n\n<p>For Xi modelling you will need to import it with Xi.ImportDataAsync</p>\n\n<p>Cheers</p>\n\n<p>Adam</p>",
      "rawMarkdown": "Hi everyone,\r\n\r\nFeature engineering in F# is a lot of fun so I decided to create more \"magic\" features, this time based on duplicates.\r\n\r\nI define a duplicate in 3 ways:\r\n\r\n1. All numeric features are the same \r\n\r\n2. All date features are the same\r\n\r\n3. Both numeric and date features are the same\r\n\r\nI calculate hashcodes of the features and then simply group by the hashcode to find groups of identical rows across train and test.\r\n\r\nFor each train and test record 3 features are then engineered:\r\n\r\n1. Number of duplicates in the duplicate group which contains the record. For most records this will be 1.\r\n\r\n2. Boolean flag which is True if this group has consecutive Ids (\"series\" - see my previous post)\r\n\r\n3. Number of faults in the group as int option - response of the record. It turns out if we know that a record has a duplicate that is faulty then the probability of it being faulty as well is about 38%:)\r\n\r\nThe script is here:\r\nhttps://github.com/Statfactory/Kaggle_Bosch\r\n\r\nThe script first creates a csv file, so you can use it in R or Python as well.\r\n\r\nFor Xi modelling you will need to import it with Xi.ImportDataAsync\r\n\r\nCheers\r\n\r\nAdam",
      "votes": null
    },
    {
      "id": "143449",
      "postDate": "11/08/2016 21:09:42",
      "content": "<p>Excellent! Thanks for sharing this.</p>",
      "rawMarkdown": "Excellent! Thanks for sharing this.",
      "votes": null
    },
    {
      "id": "143450",
      "postDate": "11/08/2016 21:11:01",
      "content": "<p>You're welcome:)</p>",
      "rawMarkdown": "You're welcome:)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 143449,
      "author_name": "triparc",
      "author_url": "",
      "post_date": "11/08/2016 21:09:42",
      "content": "<p>Excellent! Thanks for sharing this.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143450,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/08/2016 21:11:01",
      "content": "<p>You're welcome:)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "143444": "Hi everyone,\r\n\r\nFeature engineering in F# is a lot of fun so I decided to create more \"magic\" features, this time based on duplicates.\r\n\r\nI define a duplicate in 3 ways:\r\n\r\n1. All numeric features are the same \r\n\r\n2. All date features are the same\r\n\r\n3. Both numeric and date features are the same\r\n\r\nI calculate hashcodes of the features and then simply group by the hashcode to find groups of identical rows across train and test.\r\n\r\nFor each train and test record 3 features are then engineered:\r\n\r\n1. Number of duplicates in the duplicate group which contains the record. For most records this will be 1.\r\n\r\n2. Boolean flag which is True if this group has consecutive Ids (\"series\" - see my previous post)\r\n\r\n3. Number of faults in the group as int option - response of the record. It turns out if we know that a record has a duplicate that is faulty then the probability of it being faulty as well is about 38%:)\r\n\r\nThe script is here:\r\nhttps://github.com/Statfactory/Kaggle_Bosch\r\n\r\nThe script first creates a csv file, so you can use it in R or Python as well.\r\n\r\nFor Xi modelling you will need to import it with Xi.ImportDataAsync\r\n\r\nCheers\r\n\r\nAdam",
    "143449": "Excellent! Thanks for sharing this.",
    "143450": "You're welcome:)"
  },
  "source": "meta"
}