{
  "id": 353410,
  "title": "Missing horse data",
  "url": "/competitions/big-data-derby-2022/discussion/353410",
  "author_name": "",
  "post_date": "2022-09-18T09:32:53.754113900Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hello,<br>\nWe are a team of AI and ML students with Leeds university in the UK. We've started looking at the dataset and found several topics to raise with you. Many of them concern the position data and we'll compile a list once the data exploration is complete.</p>\n<p>Another question that has come up is missing data. There is a very large number of horses that are listed in official race cards but no not exist in the data. This may seem like a naive question. Maybe these horses did not finish. But their absence from the data set could cause imbalance or bias in the results. </p>\n<p>For example, Race #6 on January 1, 2019, at Aqueduct, had the following list of 10 starters:</p>\n<p>Programme     Horse<br>\n1    D Emcee<br>\n2    Dr. Lloyd<br>\n3    I Saw It All<br>\n4    Wisecrack<br>\n5    Lettercarrierhenry<br>\n6    Stone Breaker<br>\n7    Papa Jim<br>\n8    Straphanger<br>\n9    Simbasalltrouble<br>\n10    Dollarsforpennies</p>\n<p>The supplied dataset does not contain any trace of horses #1, #3, #4 for this race. </p>\n<p>Of the 14 915 starting horses in the dataset, there are an additional 3 119 horses, that were listed on the race cards, that are missing. It could be useful to know why. </p>\n<p>Did these horses:</p>\n<ul>\n<li>Not make it to the start line?</li>\n<li>Not leave the start line?</li>\n<li>Most importantly, did they start but not complete the distance? </li>\n</ul>\n<p>Is there another source of data that contains the fate of the horses missing in the dataset?</p>\n<p>Many thanks,</p>\n<p>Oliver</p>",
  "messages": [
    {
      "id": "1944395",
      "postDate": "09/18/2022 09:32:53",
      "content": "<p>Hello,<br>\nWe are a team of AI and ML students with Leeds university in the UK. We've started looking at the dataset and found several topics to raise with you. Many of them concern the position data and we'll compile a list once the data exploration is complete.</p>\n<p>Another question that has come up is missing data. There is a very large number of horses that are listed in official race cards but no not exist in the data. This may seem like a naive question. Maybe these horses did not finish. But their absence from the data set could cause imbalance or bias in the results. </p>\n<p>For example, Race #6 on January 1, 2019, at Aqueduct, had the following list of 10 starters:</p>\n<p>Programme     Horse<br>\n1    D Emcee<br>\n2    Dr. Lloyd<br>\n3    I Saw It All<br>\n4    Wisecrack<br>\n5    Lettercarrierhenry<br>\n6    Stone Breaker<br>\n7    Papa Jim<br>\n8    Straphanger<br>\n9    Simbasalltrouble<br>\n10    Dollarsforpennies</p>\n<p>The supplied dataset does not contain any trace of horses #1, #3, #4 for this race. </p>\n<p>Of the 14 915 starting horses in the dataset, there are an additional 3 119 horses, that were listed on the race cards, that are missing. It could be useful to know why. </p>\n<p>Did these horses:</p>\n<ul>\n<li>Not make it to the start line?</li>\n<li>Not leave the start line?</li>\n<li>Most importantly, did they start but not complete the distance? </li>\n</ul>\n<p>Is there another source of data that contains the fate of the horses missing in the dataset?</p>\n<p>Many thanks,</p>\n<p>Oliver</p>",
      "rawMarkdown": "Hello,\nWe are a team of AI and ML students with Leeds university in the UK. We've started looking at the dataset and found several topics to raise with you. Many of them concern the position data and we'll compile a list once the data exploration is complete.\n\nAnother question that has come up is missing data. There is a very large number of horses that are listed in official race cards but no not exist in the data. This may seem like a naive question. Maybe these horses did not finish. But their absence from the data set could cause imbalance or bias in the results. \n\nFor example, Race #6 on January 1, 2019, at Aqueduct, had the following list of 10 starters:\n\nProgramme \tHorse\n1\tD Emcee\n2\tDr. Lloyd\n3\tI Saw It All\n4\tWisecrack\n5\tLettercarrierhenry\n6\tStone Breaker\n7\tPapa Jim\n8\tStraphanger\n9\tSimbasalltrouble\n10\tDollarsforpennies\n\nThe supplied dataset does not contain any trace of horses #1, #3, #4 for this race. \n\nOf the 14 915 starting horses in the dataset, there are an additional 3 119 horses, that were listed on the race cards, that are missing. It could be useful to know why. \n\nDid these horses:\n-\tNot make it to the start line?\n-\tNot leave the start line?\n-\tMost importantly, did they start but not complete the distance? \n\nIs there another source of data that contains the fate of the horses missing in the dataset?\n\nMany thanks,\n\nOliver",
      "votes": null
    },
    {
      "id": "1944680",
      "postDate": "09/18/2022 13:54:55",
      "content": "<p>Hello Oliver:</p>\n<p>Those horses were scratches that never made the race.  Horses can be scratched for a variety of reasons from the time of the race draw to the start of the race e.g. the trainer pulls them from a race, the veterinarian rules the horse unfit to race, to the jockey feeling something is \"off\" with the horse during warm-ups.</p>\n<p>In the US, the official records of races are maintained by Equibase.  You can look up historical charts at equibase.com --&gt; Results --&gt; Historical Charts. </p>\n<p>I spot checked the data and horses who started the race but did not finish are included in the dataset--they can be identified by having a max(trakus_index) lower than the competitors who finished.</p>\n<p>Hope this helps,<br>\nBrian</p>",
      "rawMarkdown": "Hello Oliver:\n\nThose horses were scratches that never made the race.  Horses can be scratched for a variety of reasons from the time of the race draw to the start of the race e.g. the trainer pulls them from a race, the veterinarian rules the horse unfit to race, to the jockey feeling something is \"off\" with the horse during warm-ups.\n\nIn the US, the official records of races are maintained by Equibase.  You can look up historical charts at equibase.com --> Results --> Historical Charts. \n\nI spot checked the data and horses who started the race but did not finish are included in the dataset--they can be identified by having a max(trakus_index) lower than the competitors who finished.\n\nHope this helps,\nBrian",
      "votes": null
    },
    {
      "id": "1945951",
      "postDate": "09/19/2022 14:20:29",
      "content": "<p>I couldn't explain it much better than this. Horses get scratched for all sorts of reasons - competition tougher than expected, rain causes a race to switch from a grass to dirt surface, veterinary concern, etc.</p>",
      "rawMarkdown": "I couldn't explain it much better than this. Horses get scratched for all sorts of reasons - competition tougher than expected, rain causes a race to switch from a grass to dirt surface, veterinary concern, etc.",
      "votes": null
    },
    {
      "id": "1945961",
      "postDate": "09/19/2022 14:26:47",
      "content": "<p>Brian and Joe, <br>\nThank you very much for taking the time to reply. Your replies are very helpful.<br>\nBest,<br>\nOliver</p>",
      "rawMarkdown": "Brian and Joe, \nThank you very much for taking the time to reply. Your replies are very helpful.\nBest,\nOliver",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1944680,
      "author_name": "btimoney",
      "author_url": "",
      "post_date": "09/18/2022 13:54:55",
      "content": "<p>Hello Oliver:</p>\n<p>Those horses were scratches that never made the race.  Horses can be scratched for a variety of reasons from the time of the race draw to the start of the race e.g. the trainer pulls them from a race, the veterinarian rules the horse unfit to race, to the jockey feeling something is \"off\" with the horse during warm-ups.</p>\n<p>In the US, the official records of races are maintained by Equibase.  You can look up historical charts at equibase.com --&gt; Results --&gt; Historical Charts. </p>\n<p>I spot checked the data and horses who started the race but did not finish are included in the dataset--they can be identified by having a max(trakus_index) lower than the competitors who finished.</p>\n<p>Hope this helps,<br>\nBrian</p>",
      "votes": null,
      "replies": [
        {
          "id": 1945951,
          "author_name": "nythanyra",
          "author_url": "",
          "post_date": "09/19/2022 14:20:29",
          "content": "<p>I couldn't explain it much better than this. Horses get scratched for all sorts of reasons - competition tougher than expected, rain causes a race to switch from a grass to dirt surface, veterinary concern, etc.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1945961,
      "author_name": "barebackpenguin",
      "author_url": "",
      "post_date": "09/19/2022 14:26:47",
      "content": "<p>Brian and Joe, <br>\nThank you very much for taking the time to reply. Your replies are very helpful.<br>\nBest,<br>\nOliver</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1944395": "Hello,\nWe are a team of AI and ML students with Leeds university in the UK. We've started looking at the dataset and found several topics to raise with you. Many of them concern the position data and we'll compile a list once the data exploration is complete.\n\nAnother question that has come up is missing data. There is a very large number of horses that are listed in official race cards but no not exist in the data. This may seem like a naive question. Maybe these horses did not finish. But their absence from the data set could cause imbalance or bias in the results. \n\nFor example, Race #6 on January 1, 2019, at Aqueduct, had the following list of 10 starters:\n\nProgramme \tHorse\n1\tD Emcee\n2\tDr. Lloyd\n3\tI Saw It All\n4\tWisecrack\n5\tLettercarrierhenry\n6\tStone Breaker\n7\tPapa Jim\n8\tStraphanger\n9\tSimbasalltrouble\n10\tDollarsforpennies\n\nThe supplied dataset does not contain any trace of horses #1, #3, #4 for this race. \n\nOf the 14 915 starting horses in the dataset, there are an additional 3 119 horses, that were listed on the race cards, that are missing. It could be useful to know why. \n\nDid these horses:\n-\tNot make it to the start line?\n-\tNot leave the start line?\n-\tMost importantly, did they start but not complete the distance? \n\nIs there another source of data that contains the fate of the horses missing in the dataset?\n\nMany thanks,\n\nOliver",
    "1944680": "Hello Oliver:\n\nThose horses were scratches that never made the race.  Horses can be scratched for a variety of reasons from the time of the race draw to the start of the race e.g. the trainer pulls them from a race, the veterinarian rules the horse unfit to race, to the jockey feeling something is \"off\" with the horse during warm-ups.\n\nIn the US, the official records of races are maintained by Equibase.  You can look up historical charts at equibase.com --> Results --> Historical Charts. \n\nI spot checked the data and horses who started the race but did not finish are included in the dataset--they can be identified by having a max(trakus_index) lower than the competitors who finished.\n\nHope this helps,\nBrian",
    "1945951": "I couldn't explain it much better than this. Horses get scratched for all sorts of reasons - competition tougher than expected, rain causes a race to switch from a grass to dirt surface, veterinary concern, etc.",
    "1945961": "Brian and Joe, \nThank you very much for taking the time to reply. Your replies are very helpful.\nBest,\nOliver"
  },
  "source": "meta"
}