An example of where this would be a concern would be a scatter plot with points at (1, 1), (1, 2), (2,1), (2, 2). Consider the following predictions which are just a reordering of the points:
\nDo these predictions get the same score? Is there some criteria we are supposed to use to determine the desired order?
\nFor many charts there is a natural order, but this is not necessarily true for scatter plots.
", "messages": [ { "id": "2197885", "postDate": "03/26/2023 14:16:55", "content": "An example of where this would be a concern would be a scatter plot with points at (1, 1), (1, 2), (2,1), (2, 2). Consider the following predictions which are just a reordering of the points:
\nDo these predictions get the same score? Is there some criteria we are supposed to use to determine the desired order?
\nFor many charts there is a natural order, but this is not necessarily true for scatter plots.
", "rawMarkdown": "An example of where this would be a concern would be a scatter plot with points at (1, 1), (1, 2), (2,1), (2, 2). Consider the following predictions which are just a reordering of the points:\n- x-series of [1, 1, 2, 2] and y-series of [1, 2, 1, 2]\n- x-series of [1, 1, 2, 2] and y-series of [2, 1, 2, 1]\n\nDo these predictions get the same score? Is there some criteria we are supposed to use to determine the desired order?\n\nFor many charts there is a natural order, but this is not necessarily true for scatter plots.", "votes": null }, { "id": "2199099", "postDate": "03/27/2023 14:01:59", "content": "@pabyrnes Great question. This wasn't addressed sufficiently in the documentation. Generally, you should take points from left to right on the x-axis with the corresponding y-values. For your scatter plot example, let me double check if there's an ambiguity possible. If so, I may need to make a data update.
", "rawMarkdown": "pabyrnes Great question. This wasn't addressed sufficiently in the documentation. Generally, you should take points from left to right on the x-axis with the corresponding y-values. For your scatter plot example, let me double check if there's an ambiguity possible. If so, I may need to make a data update.", "votes": null }, { "id": "2203478", "postDate": "03/30/2023 20:47:30", "content": "I noticed the ground truth for training annotation file 039d3e82ebaf.json does not list points from left to right (or from top-to-bottom). The relevant part of the json is:
\n\"data-series\": [{\"x\": 46, \"y\": 10}, {\"x\": 82, \"y\": 9}, {\"x\": 83, \"y\": 9}, {\"x\": 90, \"y\": 9}, {\"x\": 132, \"y\": 3}, {\"x\": 114, \"y\": 23}, {\"x\": 123, \"y\": 47}, {\"x\": 647, \"y\": 79}, {\"x\": 641, \"y\": 87}, {\"x\": 725, \"y\": 77}]
Thanks for the heads up. I'll make an update to regularize this.
", "rawMarkdown": "Thanks for the heads up. I'll make an update to regularize this.", "votes": null }, { "id": "2203541", "postDate": "03/30/2023 22:44:06", "content": "For what it's worth even if the ground truth points are ordered this could still be an issue for nearly the same x-values. Imagine a ground truth like: \"data-series\": [{\"x\": 50, \"y\": 100}, {\"x\": 50.01, \"y\": -100}]. A prediction of \"data-series\": [{\"x\": 50, \"y\": -100}, {\"x\": 50.01, \"y\": 100}] gets a score for the y-series of 0.24. While a prediction of something like \"data-series\": [{\"x\": 50, \"y\": 0}, {\"x\": 50.01, \"y\": 0}] gets a score for the y-series of 0.54. I need to absolutely check the math, but I suspect that if there was an equal chance of the y-values being in either order guessing the average of the 2 y-values gives the maximum expected score.
I am guessing this isn't that big of an issue for overall scores (unless there are a lot of scatter plots with most of the points having nearly the same x-values and a wide range of y-values), but posting just so it is a noted issue. I will probably also background-think about if there is an efficient way to determine the ordering of the points in a prediction that gives the best score.
", "rawMarkdown": "For what it's worth even if the ground truth points are ordered this could still be an issue for nearly the same x-values. Imagine a ground truth like: `\"data-series\": [{\"x\": 50, \"y\": 100}, {\"x\": 50.01, \"y\": -100}]`. A prediction of `\"data-series\": [{\"x\": 50, \"y\": -100}, {\"x\": 50.01, \"y\": 100}]` gets a score for the y-series of `0.24`. While a prediction of something like `\"data-series\": [{\"x\": 50, \"y\": 0}, {\"x\": 50.01, \"y\": 0}]` gets a score for the y-series of `0.54`. I need to absolutely check the math, but I suspect that if there was an equal chance of the y-values being in either order guessing the average of the 2 y-values gives the maximum expected score.\n\nI am guessing this isn't that big of an issue for overall scores (unless there are a lot of scatter plots with most of the points having nearly the same x-values and a wide range of y-values), but posting just so it is a noted issue. I will probably also background-think about if there is an efficient way to determine the ordering of the points in a prediction that gives the best score.", "votes": null }, { "id": "2239487", "postDate": "04/29/2023 14:54:35", "content": "Should we take points from top to bottom for a horizontal bar graph? @ryanholbrook
", "rawMarkdown": "Should we take points from top to bottom for a horizontal bar graph? @ryanholbrook", "votes": null }, { "id": "2306525", "postDate": "06/17/2023 11:34:06", "content": "I have the same question but no answer!
", "rawMarkdown": "I have the same question but no answer!", "votes": null } ], "comments": [ { "id": 2199099, "author_name": "ryanholbrook", "author_url": "", "post_date": "03/27/2023 14:01:59", "content": "@pabyrnes Great question. This wasn't addressed sufficiently in the documentation. Generally, you should take points from left to right on the x-axis with the corresponding y-values. For your scatter plot example, let me double check if there's an ambiguity possible. If so, I may need to make a data update.
", "votes": null, "replies": [ { "id": 2203478, "author_name": "pabyrnes", "author_url": "", "post_date": "03/30/2023 20:47:30", "content": "I noticed the ground truth for training annotation file 039d3e82ebaf.json does not list points from left to right (or from top-to-bottom). The relevant part of the json is:
\n\"data-series\": [{\"x\": 46, \"y\": 10}, {\"x\": 82, \"y\": 9}, {\"x\": 83, \"y\": 9}, {\"x\": 90, \"y\": 9}, {\"x\": 132, \"y\": 3}, {\"x\": 114, \"y\": 23}, {\"x\": 123, \"y\": 47}, {\"x\": 647, \"y\": 79}, {\"x\": 641, \"y\": 87}, {\"x\": 725, \"y\": 77}]
Thanks for the heads up. I'll make an update to regularize this.
", "votes": null, "replies": [ { "id": 2203541, "author_name": "pabyrnes", "author_url": "", "post_date": "03/30/2023 22:44:06", "content": "For what it's worth even if the ground truth points are ordered this could still be an issue for nearly the same x-values. Imagine a ground truth like: \"data-series\": [{\"x\": 50, \"y\": 100}, {\"x\": 50.01, \"y\": -100}]. A prediction of \"data-series\": [{\"x\": 50, \"y\": -100}, {\"x\": 50.01, \"y\": 100}] gets a score for the y-series of 0.24. While a prediction of something like \"data-series\": [{\"x\": 50, \"y\": 0}, {\"x\": 50.01, \"y\": 0}] gets a score for the y-series of 0.54. I need to absolutely check the math, but I suspect that if there was an equal chance of the y-values being in either order guessing the average of the 2 y-values gives the maximum expected score.
I am guessing this isn't that big of an issue for overall scores (unless there are a lot of scatter plots with most of the points having nearly the same x-values and a wide range of y-values), but posting just so it is a noted issue. I will probably also background-think about if there is an efficient way to determine the ordering of the points in a prediction that gives the best score.
", "votes": null, "replies": [] } ] } ] }, { "id": 2239487, "author_name": "zhugds", "author_url": "", "post_date": "04/29/2023 14:54:35", "content": "Should we take points from top to bottom for a horizontal bar graph? @ryanholbrook
", "votes": null, "replies": [ { "id": 2306525, "author_name": "hamzahabduljalil", "author_url": "", "post_date": "06/17/2023 11:34:06", "content": "I have the same question but no answer!
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2197885": "An example of where this would be a concern would be a scatter plot with points at (1, 1), (1, 2), (2,1), (2, 2). Consider the following predictions which are just a reordering of the points:\n- x-series of [1, 1, 2, 2] and y-series of [1, 2, 1, 2]\n- x-series of [1, 1, 2, 2] and y-series of [2, 1, 2, 1]\n\nDo these predictions get the same score? Is there some criteria we are supposed to use to determine the desired order?\n\nFor many charts there is a natural order, but this is not necessarily true for scatter plots.", "2199099": "pabyrnes Great question. This wasn't addressed sufficiently in the documentation. Generally, you should take points from left to right on the x-axis with the corresponding y-values. For your scatter plot example, let me double check if there's an ambiguity possible. If so, I may need to make a data update.", "2203478": "I noticed the ground truth for training annotation file 039d3e82ebaf.json does not list points from left to right (or from top-to-bottom). The relevant part of the json is:\n`\"data-series\": [{\"x\": 46, \"y\": 10}, {\"x\": 82, \"y\": 9}, {\"x\": 83, \"y\": 9}, {\"x\": 90, \"y\": 9}, {\"x\": 132, \"y\": 3}, {\"x\": 114, \"y\": 23}, {\"x\": 123, \"y\": 47}, {\"x\": 647, \"y\": 79}, {\"x\": 641, \"y\": 87}, {\"x\": 725, \"y\": 77}]`", "2203487": "Thanks for the heads up. I'll make an update to regularize this.", "2203541": "For what it's worth even if the ground truth points are ordered this could still be an issue for nearly the same x-values. Imagine a ground truth like: `\"data-series\": [{\"x\": 50, \"y\": 100}, {\"x\": 50.01, \"y\": -100}]`. A prediction of `\"data-series\": [{\"x\": 50, \"y\": -100}, {\"x\": 50.01, \"y\": 100}]` gets a score for the y-series of `0.24`. While a prediction of something like `\"data-series\": [{\"x\": 50, \"y\": 0}, {\"x\": 50.01, \"y\": 0}]` gets a score for the y-series of `0.54`. I need to absolutely check the math, but I suspect that if there was an equal chance of the y-values being in either order guessing the average of the 2 y-values gives the maximum expected score.\n\nI am guessing this isn't that big of an issue for overall scores (unless there are a lot of scatter plots with most of the points having nearly the same x-values and a wide range of y-values), but posting just so it is a noted issue. I will probably also background-think about if there is an efficient way to determine the ordering of the points in a prediction that gives the best score.", "2239487": "Should we take points from top to bottom for a horizontal bar graph? @ryanholbrook", "2306525": "I have the same question but no answer!" }, "source": "meta" }