{ "id": 346864, "title": "Adding scraped data on horse identity and finishing position", "url": "/competitions/big-data-derby-2022/discussion/346864", "author_name": "Mark Green", "post_date": "2022-08-21T19:55:36.190000", "votes": 32, "comment_count": 17, "views": 0, "content": "

The current data set does not include any information on
\na) the unique identity of the horse (only its program number in an individial race)
\nb) its finishing position

\n

But both of these datasets are available from the NYRA website. Is it acceptable to add this data in to an entry, or are we restricted to only the datasets provided? a) is particularly useful for potentially tracking a horse's performance over multiple races.

", "messages": [ { "id": 1908563, "postDate": "2022-08-21T19:55:36.190Z", "content": "

The current data set does not include any information on
\na) the unique identity of the horse (only its program number in an individial race)
\nb) its finishing position

\n

But both of these datasets are available from the NYRA website. Is it acceptable to add this data in to an entry, or are we restricted to only the datasets provided? a) is particularly useful for potentially tracking a horse's performance over multiple races.

", "rawMarkdown": "The current data set does not include any information on\na) the unique identity of the horse (only its program number in an individial race)\nb) its finishing position\n\nBut both of these datasets are available from the NYRA website. Is it acceptable to add this data in to an entry, or are we restricted to only the datasets provided? a) is particularly useful for potentially tracking a horse's performance over multiple races.\n", "votes": 32 }, { "id": 1909521, "postDate": "2022-08-22T17:03:51.167Z", "content": "

Uploaded the dataset. https://www.kaggle.com/datasets/themarkgreen/big-data-derby-2022-global-horse-ids-and-places

", "rawMarkdown": "Uploaded the dataset. https://www.kaggle.com/datasets/themarkgreen/big-data-derby-2022-global-horse-ids-and-places", "votes": 10, "replies": [ { "id": 1909545, "postDate": "2022-08-22T17:32:08.690Z", "content": "

Thank you for sharing the dataset🙌

", "rawMarkdown": "Thank you for sharing the dataset🙌", "votes": 1 }, { "id": 1912370, "postDate": "2022-08-24T17:16:01.413Z", "content": "

@themarkgreen you are really an MVP. I was only lacking a Horse's Name for each race as we usually bet on the horse and we can leverage this horse's name in multiple scenarios. Thank you for providing such great Data. I appreciate it, it saved me a whole lot of time also would appreciate it if anyone can share the age detail of the horse so we make a custom probability function.

", "rawMarkdown": "@themarkgreen you are really an MVP. I was only lacking a Horse's Name for each race as we usually bet on the horse and we can leverage this horse's name in multiple scenarios. Thank you for providing such great Data. I appreciate it, it saved me a whole lot of time also would appreciate it if anyone can share the age detail of the horse so we make a custom probability function.", "votes": 1 } ] }, { "id": 1910094, "postDate": "2022-08-23T07:02:43.307Z", "content": "

@themarkgreen - thank you very much for sharing the data!

\n

There are some interesting findings within this dataset.

\n\n

\"\"

", "rawMarkdown": "@themarkgreen - thank you very much for sharing the data!\n\nThere are some interesting findings within this dataset.\n- There are up to 11 different jockeys for a single horse (horse ID 1207).\n- There are up to 780 different horses (!) for a single jockey (Manuel Franco who competed in 1274 races out of the 2000 races in the original dataset).\n- The largest number of different races for a horse is 21 (horse ID 402).\n- No horse raced in more than one race in a single day.\n- No jockey raced in more than one track ID (AQU, BEL, SAR) in a single day.\n- The largest number of races in a single day for a jockey is 12.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6272010%2F1db302c690394e0664de8761dfbff1ae%2Fplot.png?generation=1661237861191819&alt=media)", "votes": 7 }, { "id": 1996289, "postDate": "2022-10-20T07:03:07.463Z", "content": "

Thanks a lot for this enriched dataset.
\nI wonder if we can use such a data, or if we are supposed to get results only from the original dataset.

\n

@nythanyra any advice about this ?

\n

Thanks !

", "rawMarkdown": "Thanks a lot for this enriched dataset.\nI wonder if we can use such a data, or if we are supposed to get results only from the original dataset.\n\n@nythanyra any advice about this ?\n\nThanks !", "votes": 1 }, { "id": 1988774, "postDate": "2022-10-15T14:35:01.357Z", "content": "

Hi Mark, Thank you for the dataset.

\n

After exploring the horse_id dataset, I have some questions.

\n\n
Shape of nyra_start_table: (14915, 8)\nShape of nyra_race_table: (2000, 10)\nShape of horse_table: (14916, 7)\n
\n\n
track_id            0\nrace_date           0\nrace_number         0\nprogram_number      0\nweight_carried      0\njockey              0\nodds                0\nfinal_place         0\nUnnamed: 0        103\nhorse_id          103\n
\n

Here is my notebook: bdd_horse_id_check
\nCan anyone check the notebook? If I have done everything correct.
\nLet me know If anything is wrong with the code.

", "rawMarkdown": "Hi Mark, Thank you for the dataset.\n\nAfter exploring the horse_id dataset, I have some questions.\n- There is 1 row more in the horse_id dataset compared to start_table\n```\nShape of nyra_start_table: (14915, 8)\nShape of nyra_race_table: (2000, 10)\nShape of horse_table: (14916, 7)\n```\n- Data is not matching 103 rows. I used `['track_id','race_date','race_number','program_number','final_place']`\n```\ntrack_id 0\nrace_date 0\nrace_number 0\nprogram_number 0\nweight_carried 0\njockey 0\nodds 0\nfinal_place 0\nUnnamed: 0 103\nhorse_id 103\n```\nHere is my notebook: [bdd_horse_id_check](https://www.kaggle.com/asksawant/bdd-horse-id-check)\nCan anyone check the notebook? If I have done everything correct.\nLet me know If anything is wrong with the code.", "votes": 1, "replies": [ { "id": 2014085, "postDate": "2022-11-02T09:55:11.620Z", "content": "

Hi, i've noticed the dataset with the horse_ids dataset has 14916 rows while start_table dataset has only 14916. The problem is with the race [(\"track_id\" ==BEL) & (\"race_date\" ==2019-05-27) & (race_number==9)], in start_table dataset, this race has the next program_numbers --> [1, 2, 3, 4, 5, 6, 7, 8] but in the horse_ids data there is an extra \"10\" program_number. I've looked the race in https://www.nyra.com/belmont/racing/results?date=2019-05-27 and there is a horse with the number of 10 called Stone Breaker so horse_ids data is correct but i've also notices that the 10 horse is in \"scraches\" section and if you go to https://www.nyra.com/belmont/racing/results?date=2019-05-12 for example and to the race number 1 and compare to the data in start_table dataset for this race you can see that the horses in the \"scraches\" section isn't added to the data, but in the race mentioned above, the horse is in both \"also run\" section and \"scratches\" section. I'm not sure what's exactly means the \"scratches\" section, maybe it only refers to any wounds and thats the reason why the 10 horse is un both sections but, in my opinion, if in the start_table data isn't the 10 horse, I'd drop that row.

", "rawMarkdown": "Hi, i've noticed the dataset with the horse_ids dataset has 14916 rows while start_table dataset has only 14916. The problem is with the race [(\"track_id\" ==BEL) & (\"race_date\" ==2019-05-27) & (race_number==9)], in start_table dataset, this race has the next program_numbers --> [1, 2, 3, 4, 5, 6, 7, 8] but in the horse_ids data there is an extra \"10\" program_number. I've looked the race in https://www.nyra.com/belmont/racing/results?date=2019-05-27 and there is a horse with the number of 10 called Stone Breaker so horse_ids data is correct but i've also notices that the 10 horse is in \"scraches\" section and if you go to https://www.nyra.com/belmont/racing/results?date=2019-05-12 for example and to the race number 1 and compare to the data in start_table dataset for this race you can see that the horses in the \"scraches\" section isn't added to the data, but in the race mentioned above, the horse is in both \"also run\" section and \"scratches\" section. I'm not sure what's exactly means the \"scratches\" section, maybe it only refers to any wounds and thats the reason why the 10 horse is un both sections but, in my opinion, if in the start_table data isn't the 10 horse, I'd drop that row." }, { "id": 2014094, "postDate": "2022-11-02T10:13:42.383Z", "content": "

And the 103 rows not matching maybe is because the \"program_number\" column has other values

", "rawMarkdown": "And the 103 rows not matching maybe is because the \"program_number\" column has other values" }, { "id": 2014389, "postDate": "2022-11-02T14:29:12.047Z", "content": "

A scratch is when a horse has been entered for a race, but is subsequently declared out of that race. Scratches are typically announced (and posted) the morning of race day. They can occur for many reasons (e.g. change in weather, horse not feeling well). On occassion, they occur immediately prior to the race.

", "rawMarkdown": "A scratch is when a horse has been entered for a race, but is subsequently declared out of that race. Scratches are typically announced (and posted) the morning of race day. They can occur for many reasons (e.g. change in weather, horse not feeling well). On occassion, they occur immediately prior to the race." } ] }, { "id": 1909031, "postDate": "2022-08-22T08:28:49.657Z", "content": "

Please make a dataset with this info. I cannot access the nyra webpage because the EU GDPR regulations so that nyra was not interested to add all these extra costs for the few EU visitors.

", "rawMarkdown": "Please make a dataset with this info. I cannot access the nyra webpage because the EU GDPR regulations so that nyra was not interested to add all these extra costs for the few EU visitors. ", "votes": 1 }, { "id": 1908988, "postDate": "2022-08-22T07:37:17.363Z", "content": "

From the competition rules 7C:

\n
\n

External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available to use by all participants of the Competition for purposes of the competition at no cost to the other participants.

\n
", "rawMarkdown": "From the competition rules 7C:\n\n> External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available to use by all participants of the Competition for purposes of the competition at no cost to the other participants.", "votes": 1, "replies": [ { "id": 1909032, "postDate": "2022-08-22T08:29:35.897Z", "content": "

Can you access the nyra webpage? I tried but the site is blocked on my site, i guess because of the EU gdpr. Maybe you used a vpn.

", "rawMarkdown": "Can you access the nyra webpage? I tried but the site is blocked on my site, i guess because of the EU gdpr. Maybe you used a vpn. \n", "votes": 1 }, { "id": 1909471, "postDate": "2022-08-22T16:23:02.400Z", "content": "

I can access all nyra sites without a vpn. This is just a guess, but could it be blocked in your country because of some gambling related laws?

", "rawMarkdown": "I can access all nyra sites without a vpn. This is just a guess, but could it be blocked in your country because of some gambling related laws?", "votes": 2 }, { "id": 1957991, "postDate": "2022-09-27T08:40:12.547Z", "content": "

I also can not access the nyra webpage. maybe due to some gambling-related laws

", "rawMarkdown": "I also can not access the nyra webpage. maybe due to some gambling-related laws" } ] }, { "id": 1910952, "postDate": "2022-08-23T19:58:52.610Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 1912541, "postDate": "2022-08-24T19:50:53.297Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1987651, "postDate": "2022-10-14T19:19:47.013Z", "content": "

Really, thank you very much!

", "rawMarkdown": "Really, thank you very much!" } ], "comments": [ { "id": 1909521, "author_name": "Mark Green", "author_url": "", "post_date": "2022-08-22T17:03:51.167000", "content": "

Uploaded the dataset. https://www.kaggle.com/datasets/themarkgreen/big-data-derby-2022-global-horse-ids-and-places

", "votes": 10, "replies": [ { "id": 1909545, "author_name": "Lennart Haupts", "author_url": "", "post_date": "2022-08-22T17:32:08.690000", "content": "

Thank you for sharing the dataset🙌

", "votes": 1, "replies": [] }, { "id": 1912370, "author_name": "Kush Trivedi", "author_url": "", "post_date": "2022-08-24T17:16:01.413000", "content": "

@themarkgreen you are really an MVP. I was only lacking a Horse's Name for each race as we usually bet on the horse and we can leverage this horse's name in multiple scenarios. Thank you for providing such great Data. I appreciate it, it saved me a whole lot of time also would appreciate it if anyone can share the age detail of the horse so we make a custom probability function.

", "votes": 1, "replies": [] } ] }, { "id": 1910094, "author_name": "shapital", "author_url": "", "post_date": "2022-08-23T07:02:43.307000", "content": "

@themarkgreen - thank you very much for sharing the data!

\n

There are some interesting findings within this dataset.

\n\n

\"\"

", "votes": 7, "replies": [] }, { "id": 1996289, "author_name": "Pete Gore", "author_url": "", "post_date": "2022-10-20T07:03:07.463000", "content": "

Thanks a lot for this enriched dataset.
\nI wonder if we can use such a data, or if we are supposed to get results only from the original dataset.

\n

@nythanyra any advice about this ?

\n

Thanks !

", "votes": 1, "replies": [] }, { "id": 1988774, "author_name": "Prathamesh Sawant", "author_url": "", "post_date": "2022-10-15T14:35:01.357000", "content": "

Hi Mark, Thank you for the dataset.

\n

After exploring the horse_id dataset, I have some questions.

\n\n
Shape of nyra_start_table: (14915, 8)\nShape of nyra_race_table: (2000, 10)\nShape of horse_table: (14916, 7)\n
\n\n
track_id            0\nrace_date           0\nrace_number         0\nprogram_number      0\nweight_carried      0\njockey              0\nodds                0\nfinal_place         0\nUnnamed: 0        103\nhorse_id          103\n
\n

Here is my notebook: bdd_horse_id_check
\nCan anyone check the notebook? If I have done everything correct.
\nLet me know If anything is wrong with the code.

", "votes": 1, "replies": [ { "id": 2014085, "author_name": "Antonio Morales", "author_url": "", "post_date": "2022-11-02T09:55:11.620000", "content": "

Hi, i've noticed the dataset with the horse_ids dataset has 14916 rows while start_table dataset has only 14916. The problem is with the race [(\"track_id\" ==BEL) & (\"race_date\" ==2019-05-27) & (race_number==9)], in start_table dataset, this race has the next program_numbers --> [1, 2, 3, 4, 5, 6, 7, 8] but in the horse_ids data there is an extra \"10\" program_number. I've looked the race in https://www.nyra.com/belmont/racing/results?date=2019-05-27 and there is a horse with the number of 10 called Stone Breaker so horse_ids data is correct but i've also notices that the 10 horse is in \"scraches\" section and if you go to https://www.nyra.com/belmont/racing/results?date=2019-05-12 for example and to the race number 1 and compare to the data in start_table dataset for this race you can see that the horses in the \"scraches\" section isn't added to the data, but in the race mentioned above, the horse is in both \"also run\" section and \"scratches\" section. I'm not sure what's exactly means the \"scratches\" section, maybe it only refers to any wounds and thats the reason why the 10 horse is un both sections but, in my opinion, if in the start_table data isn't the 10 horse, I'd drop that row.

", "votes": 0, "replies": [] }, { "id": 2014094, "author_name": "Antonio Morales", "author_url": "", "post_date": "2022-11-02T10:13:42.383000", "content": "

And the 103 rows not matching maybe is because the \"program_number\" column has other values

", "votes": 0, "replies": [] }, { "id": 2014389, "author_name": "Joe Appelbaum", "author_url": "", "post_date": "2022-11-02T14:29:12.047000", "content": "

A scratch is when a horse has been entered for a race, but is subsequently declared out of that race. Scratches are typically announced (and posted) the morning of race day. They can occur for many reasons (e.g. change in weather, horse not feeling well). On occassion, they occur immediately prior to the race.

", "votes": 0, "replies": [] } ] }, { "id": 1909031, "author_name": "Adrian Muresan", "author_url": "", "post_date": "2022-08-22T08:28:49.657000", "content": "

Please make a dataset with this info. I cannot access the nyra webpage because the EU GDPR regulations so that nyra was not interested to add all these extra costs for the few EU visitors.

", "votes": 1, "replies": [] }, { "id": 1908988, "author_name": "Lennart Haupts", "author_url": "", "post_date": "2022-08-22T07:37:17.363000", "content": "

From the competition rules 7C:

\n
\n

External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available to use by all participants of the Competition for purposes of the competition at no cost to the other participants.

\n
", "votes": 1, "replies": [ { "id": 1909032, "author_name": "Adrian Muresan", "author_url": "", "post_date": "2022-08-22T08:29:35.897000", "content": "

Can you access the nyra webpage? I tried but the site is blocked on my site, i guess because of the EU gdpr. Maybe you used a vpn.

", "votes": 1, "replies": [] }, { "id": 1909471, "author_name": "Lennart Haupts", "author_url": "", "post_date": "2022-08-22T16:23:02.400000", "content": "

I can access all nyra sites without a vpn. This is just a guess, but could it be blocked in your country because of some gambling related laws?

", "votes": 2, "replies": [] }, { "id": 1957991, "author_name": "naiborhujosua", "author_url": "", "post_date": "2022-09-27T08:40:12.547000", "content": "

I also can not access the nyra webpage. maybe due to some gambling-related laws

", "votes": 0, "replies": [] } ] }, { "id": 1910952, "author_name": "", "author_url": "", "post_date": "2022-08-23T19:58:52.610000", "content": "", "votes": 1, "replies": [ { "id": 1912541, "author_name": "", "author_url": "", "post_date": "2022-08-24T19:50:53.297000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1987651, "author_name": "", "author_url": "", "post_date": "2022-10-14T19:19:47.013000", "content": "

Really, thank you very much!

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1908563": "The current data set does not include any information on\na) the unique identity of the horse (only its program number in an individial race)\nb) its finishing position\n\nBut both of these datasets are available from the NYRA website. Is it acceptable to add this data in to an entry, or are we restricted to only the datasets provided? a) is particularly useful for potentially tracking a horse's performance over multiple races.\n", "1909521": "Uploaded the dataset. https://www.kaggle.com/datasets/themarkgreen/big-data-derby-2022-global-horse-ids-and-places", "1910094": "@themarkgreen - thank you very much for sharing the data!\n\nThere are some interesting findings within this dataset.\n- There are up to 11 different jockeys for a single horse (horse ID 1207).\n- There are up to 780 different horses (!) for a single jockey (Manuel Franco who competed in 1274 races out of the 2000 races in the original dataset).\n- The largest number of different races for a horse is 21 (horse ID 402).\n- No horse raced in more than one race in a single day.\n- No jockey raced in more than one track ID (AQU, BEL, SAR) in a single day.\n- The largest number of races in a single day for a jockey is 12.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6272010%2F1db302c690394e0664de8761dfbff1ae%2Fplot.png?generation=1661237861191819&alt=media)", "1996289": "Thanks a lot for this enriched dataset.\nI wonder if we can use such a data, or if we are supposed to get results only from the original dataset.\n\n@nythanyra any advice about this ?\n\nThanks !", "1988774": "Hi Mark, Thank you for the dataset.\n\nAfter exploring the horse_id dataset, I have some questions.\n- There is 1 row more in the horse_id dataset compared to start_table\n```\nShape of nyra_start_table: (14915, 8)\nShape of nyra_race_table: (2000, 10)\nShape of horse_table: (14916, 7)\n```\n- Data is not matching 103 rows. I used `['track_id','race_date','race_number','program_number','final_place']`\n```\ntrack_id 0\nrace_date 0\nrace_number 0\nprogram_number 0\nweight_carried 0\njockey 0\nodds 0\nfinal_place 0\nUnnamed: 0 103\nhorse_id 103\n```\nHere is my notebook: [bdd_horse_id_check](https://www.kaggle.com/asksawant/bdd-horse-id-check)\nCan anyone check the notebook? If I have done everything correct.\nLet me know If anything is wrong with the code.", "1909031": "Please make a dataset with this info. I cannot access the nyra webpage because the EU GDPR regulations so that nyra was not interested to add all these extra costs for the few EU visitors. ", "1908988": "From the competition rules 7C:\n\n> External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available to use by all participants of the Competition for purposes of the competition at no cost to the other participants.", "1910952": "", "1987651": "Really, thank you very much!" } }