I have seen works with very good statistics, but they use the Haversine formula that is not suitable for this type of competition.
\nThe Haversine formula is good for measuring large distances with low computational cost.
\nI have left the code for the Vincenty formula (used in geodesy to measure millimetric movements of tectonic plates among other uses).
\nHere is the code for Vincenty's formula: copy and paste it.
\nhttps://www.kaggle.com/code/gabrielbeguerie/derby-2022-vincenty-formulae-distance
\nThe margin of error of Vincenty's formula is 5 millimeters and it works very well to calculate the true distance of the horses and their true speeds.
\nCheers
", "messages": [ { "id": 1971880, "postDate": "2022-10-04T19:57:13.410Z", "content": "Thanks for bringing up this topic, Gabriel. I also like Vincenty for calculating large distances. Here though I don't think it matters.
\nThese are small enough distances to assume the earth is flat. You should be able to use a simple Euclidean distance after converting lat/long to an x-y distance. But the more formal metrics should also work.
\nI haven't tested these points so it's really just top-of-mind thoughts.
Hi John:
\nI thought the same as you about using Eucledian coordinates since the distances are so small that I didn't think it was worth using any geographic coordinate formula (Haversine, Vincenty or Karney).
\nAfter testing I have realized that using them makes very big differences in the results. Yesterday I tested a work by @Shapital that uses Eucledian distances with the Python Geopy library (Karney method the best so far), and the results were much better with Geopy or Vincenty than with simple eucledian distances.
\nI did not know that the use or not of the formulas of Vincenty or Karney could be so important in such small distances. But it seems that if mainly to do speed analysis.
\nCheers
", "rawMarkdown": "Hi John:\n\nI thought the same as you about using Eucledian coordinates since the distances are so small that I didn't think it was worth using any geographic coordinate formula (Haversine, Vincenty or Karney).\n\nAfter testing I have realized that using them makes very big differences in the results. Yesterday I tested a work by @Shapital that uses Eucledian distances with the Python Geopy library (Karney method the best so far), and the results were much better with Geopy or Vincenty than with simple eucledian distances.\n\nI did not know that the use or not of the formulas of Vincenty or Karney could be so important in such small distances. But it seems that if mainly to do speed analysis.\n\nCheers", "votes": 1 }, { "id": 1985171, "postDate": "2022-10-13T06:48:40.517Z", "content": "same here - i thought the difference would be negligible until I saw the results using different methods.
", "rawMarkdown": "same here - i thought the difference would be negligible until I saw the results using different methods. " }, { "id": 1989279, "postDate": "2022-10-15T20:27:25.687Z", "content": "The differences at each point don't seem like much. From what I've seen, the total difference is < 0.2% of total race distance and point differences are < 1cm on average. Do you have a scenario that requires more precision?
\nNotebook: https://www.kaggle.com/code/jpmiller/vincenty-or-haversine-does-it-matter
", "rawMarkdown": "The differences at each point don't seem like much. From what I've seen, the total difference is < 0.2% of total race distance and point differences are < 1cm on average. Do you have a scenario that requires more precision?\n\nNotebook: https://www.kaggle.com/code/jpmiller/vincenty-or-haversine-does-it-matter" } ] }, { "id": 1967215, "postDate": "2022-10-02T11:03:57.137Z", "content": "I have seen works with very good statistics, but they use the Haversine formula that is not suitable for this type of competition.
\nThe Haversine formula is good for measuring large distances with low computational cost.
\nI have left the code for the Vincenty formula (used in geodesy to measure millimetric movements of tectonic plates among other uses).
\nHere is the code for Vincenty's formula: copy and paste it.
\nhttps://www.kaggle.com/code/gabrielbeguerie/derby-2022-vincenty-formulae-distance
\nThe margin of error of Vincenty's formula is 5 millimeters and it works very well to calculate the true distance of the horses and their true speeds.
\nCheers
", "rawMarkdown": "I have seen works with very good statistics, but they use the Haversine formula that is not suitable for this type of competition.\n\nThe Haversine formula is good for measuring large distances with low computational cost.\n\nI have left the code for the Vincenty formula (used in geodesy to measure millimetric movements of tectonic plates among other uses).\n\nHere is the code for Vincenty's formula: copy and paste it.\n\n[https://www.kaggle.com/code/gabrielbeguerie/derby-2022-vincenty-formulae-distance](url)\n\nThe margin of error of Vincenty's formula is 5 millimeters and it works very well to calculate the true distance of the horses and their true speeds.\n\nCheers", "votes": 4 }, { "id": 1992489, "postDate": "2022-10-17T17:38:25.097Z", "content": "Is the distance that important ? I mean it's interesting to have it precisely for a single-horse study, but in a matter of comparison, both the GPS precision and the projection are négligeable because they are the same for all horses ?
", "rawMarkdown": "Is the distance that important ? I mean it's interesting to have it precisely for a single-horse study, but in a matter of comparison, both the GPS precision and the projection are négligeable because they are the same for all horses ?", "votes": 1, "replies": [ { "id": 2005476, "postDate": "2022-10-27T03:17:18.783Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2005483, "postDate": "2022-10-27T03:34:42.933Z", "content": "I agree, Pete. One thing I recently learned is that the system uses local transmitters and receivers, which in theory would avoid some of the variation in GPS across tracks and days. I'm into it because of general interest and not out of necessity for this analysis.
\nHere's a link to the patent for anyone having trouble sleeping, 😁
\nhttps://patents.google.com/patent/US7009561/en
Thanks for bringing up this topic, Gabriel. I also like Vincenty for calculating large distances. Here though I don't think it matters.
\nThese are small enough distances to assume the earth is flat. You should be able to use a simple Euclidean distance after converting lat/long to an x-y distance. But the more formal metrics should also work.
\nI haven't tested these points so it's really just top-of-mind thoughts.
Hi John:
\nI thought the same as you about using Eucledian coordinates since the distances are so small that I didn't think it was worth using any geographic coordinate formula (Haversine, Vincenty or Karney).
\nAfter testing I have realized that using them makes very big differences in the results. Yesterday I tested a work by @Shapital that uses Eucledian distances with the Python Geopy library (Karney method the best so far), and the results were much better with Geopy or Vincenty than with simple eucledian distances.
\nI did not know that the use or not of the formulas of Vincenty or Karney could be so important in such small distances. But it seems that if mainly to do speed analysis.
\nCheers
", "votes": 1, "replies": [] }, { "id": 1985171, "author_name": "xxxxyyyy80008", "author_url": "", "post_date": "2022-10-13T06:48:40.517000", "content": "same here - i thought the difference would be negligible until I saw the results using different methods.
", "votes": 0, "replies": [] }, { "id": 1989279, "author_name": "JohnM", "author_url": "", "post_date": "2022-10-15T20:27:25.687000", "content": "The differences at each point don't seem like much. From what I've seen, the total difference is < 0.2% of total race distance and point differences are < 1cm on average. Do you have a scenario that requires more precision?
\nNotebook: https://www.kaggle.com/code/jpmiller/vincenty-or-haversine-does-it-matter
", "votes": 0, "replies": [] } ] }, { "id": 1992489, "author_name": "Pete Gore", "author_url": "", "post_date": "2022-10-17T17:38:25.097000", "content": "Is the distance that important ? I mean it's interesting to have it precisely for a single-horse study, but in a matter of comparison, both the GPS precision and the projection are négligeable because they are the same for all horses ?
", "votes": 1, "replies": [ { "id": 2005476, "author_name": "", "author_url": "", "post_date": "2022-10-27T03:17:18.783000", "content": "", "votes": 0, "replies": [] }, { "id": 2005483, "author_name": "JohnM", "author_url": "", "post_date": "2022-10-27T03:34:42.933000", "content": "I agree, Pete. One thing I recently learned is that the system uses local transmitters and receivers, which in theory would avoid some of the variation in GPS across tracks and days. I'm into it because of general interest and not out of necessity for this analysis.
\nHere's a link to the patent for anyone having trouble sleeping, 😁
\nhttps://patents.google.com/patent/US7009561/en