{
  "id": 21841,
  "title": "Geocoding the locations using R and Nominatim",
  "url": "/competitions/grupo-bimbo-inventory-demand/discussion/21841",
  "author_name": "",
  "post_date": "2016-06-22T12:16:15.650Z",
  "votes": 6,
  "comment_count": 5,
  "views": 939,
  "content": "<p>I have written an R script to clean up the location data, geocode and then plot a map. Unfortunately it won't run in Kaggle scripts because the geocoding needs an internet connection.\n<img src=\"https://s3-ap-southeast-1.amazonaws.com/tutorialfiles/map.png\" alt=\"enter image description here\" title></p>",
  "messages": [
    {
      "id": "124790",
      "postDate": "06/22/2016 12:16:15",
      "content": "<p>I have written an R script to clean up the location data, geocode and then plot a map. Unfortunately it won't run in Kaggle scripts because the geocoding needs an internet connection.\n<img src=\"https://s3-ap-southeast-1.amazonaws.com/tutorialfiles/map.png\" alt=\"enter image description here\" title></p>",
      "rawMarkdown": "I have written an R script to clean up the location data, geocode and then plot a map. Unfortunately it won't run in Kaggle scripts because the geocoding needs an internet connection.\r\n![enter image description here][1]\r\n\r\n\r\n  [1]: https://s3-ap-southeast-1.amazonaws.com/tutorialfiles/map.png",
      "votes": null
    },
    {
      "id": "124824",
      "postDate": "06/22/2016 15:55:10",
      "content": "<p>I assume that, with this information, you can take the aggregate demand in places that are very close from each other, and model that, instead of modelling each separate store. However, that will only happen when a group of stores are very close... which is when a client will go to a second store if he cannot find his product in the first one.</p>\n\n<p>I am not sure how large the improvement will be, if any, but may be worth trying.</p>",
      "rawMarkdown": "I assume that, with this information, you can take the aggregate demand in places that are very close from each other, and model that, instead of modelling each separate store. However, that will only happen when a group of stores are very close... which is when a client will go to a second store if he cannot find his product in the first one.\r\n\r\nI am not sure how large the improvement will be, if any, but may be worth trying.",
      "votes": null
    },
    {
      "id": "124852",
      "postDate": "06/22/2016 19:03:44",
      "content": "<p>HI, beware that you cannot use external data according to the competition rules.  Don't ask me why it is that way...</p>\n\n<p>I don't see how you can geocode without external data access hence I'm afraid you have to not use this in your submission.</p>",
      "rawMarkdown": "HI, beware that you cannot use external data according to the competition rules.  Don't ask me why it is that way...\r\n\r\nI don't see how you can geocode without external data access hence I'm afraid you have to not use this in your submission.",
      "votes": null
    },
    {
      "id": "124857",
      "postDate": "06/22/2016 19:30:28",
      "content": "<p>The rule against external data is fair and not that strange; the competition organizer must (at least potentially) be able to use the solution in production. And if you use external data that may be restricted by licensing issues - or that may degrade over time - that prevents them from using the solutions.</p>\n\n<p>Also, you could sit down and make predictions by hand and call those external data. That would be completely useless as a ML solution and not useful for the organizer in the future. Try to come up with a definition of what external data should be allowed, and you'll quickly realize how tricky it is. </p>\n\n<p>Now, that doesn't mean that the information provided by Colin is useless, quite the opposite. If you're able to show that the data is useful, maybe there's another way to deduce the data? Some machine learning technique to figure out which towns are close to each other and which ones are far away from each other? Develop a technique using Colins data, then remove Colins data once your technique performs as well without it - and you're in the clear. </p>\n\n<p>If you can generate a winning solution without any external data, then it doesn't matter what data you considered getting there. That's merely the process of feature engineering. </p>\n\n<p>/m</p>",
      "rawMarkdown": "The rule against external data is fair and not that strange; the competition organizer must (at least potentially) be able to use the solution in production. And if you use external data that may be restricted by licensing issues - or that may degrade over time - that prevents them from using the solutions.\r\n\r\nAlso, you could sit down and make predictions by hand and call those external data. That would be completely useless as a ML solution and not useful for the organizer in the future. Try to come up with a definition of what external data should be allowed, and you'll quickly realize how tricky it is. \r\n\r\nNow, that doesn't mean that the information provided by Colin is useless, quite the opposite. If you're able to show that the data is useful, maybe there's another way to deduce the data? Some machine learning technique to figure out which towns are close to each other and which ones are far away from each other? Develop a technique using Colins data, then remove Colins data once your technique performs as well without it - and you're in the clear. \r\n\r\nIf you can generate a winning solution without any external data, then it doesn't matter what data you considered getting there. That's merely the process of feature engineering. \r\n\r\n/m",
      "votes": null
    },
    {
      "id": "124887",
      "postDate": "06/23/2016 05:47:08",
      "content": "<p>I agree with Matthias.</p>\n\n<p>You can use the latitude and longitude to build a spacial model, then use the information from that to group locations together (maybe by city name or by state). Then your final model uses the grouped locations as features, not the latitudes and longitudes.</p>",
      "rawMarkdown": "I agree with Matthias.\r\n\r\nYou can use the latitude and longitude to build a spacial model, then use the information from that to group locations together (maybe by city name or by state). Then your final model uses the grouped locations as features, not the latitudes and longitudes.",
      "votes": null
    },
    {
      "id": "124907",
      "postDate": "06/23/2016 09:57:44",
      "content": "<p>Colin,</p>\n\n<p>you will need to explain how you grouped locations together using only the data of the competition.  If you can do so, then great.</p>\n\n<p>I also agree that using external data to better understand the problem is good.   I may look at your map in detail ;)</p>",
      "rawMarkdown": "Colin,\r\n\r\nyou will need to explain how you grouped locations together using only the data of the competition.  If you can do so, then great.\r\n\r\nI also agree that using external data to better understand the problem is good.   I may look at your map in detail ;)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 124824,
      "author_name": "manuelsh",
      "author_url": "",
      "post_date": "06/22/2016 15:55:10",
      "content": "<p>I assume that, with this information, you can take the aggregate demand in places that are very close from each other, and model that, instead of modelling each separate store. However, that will only happen when a group of stores are very close... which is when a client will go to a second store if he cannot find his product in the first one.</p>\n\n<p>I am not sure how large the improvement will be, if any, but may be worth trying.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124852,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "06/22/2016 19:03:44",
      "content": "<p>HI, beware that you cannot use external data according to the competition rules.  Don't ask me why it is that way...</p>\n\n<p>I don't see how you can geocode without external data access hence I'm afraid you have to not use this in your submission.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124857,
      "author_name": "mfagerlund",
      "author_url": "",
      "post_date": "06/22/2016 19:30:28",
      "content": "<p>The rule against external data is fair and not that strange; the competition organizer must (at least potentially) be able to use the solution in production. And if you use external data that may be restricted by licensing issues - or that may degrade over time - that prevents them from using the solutions.</p>\n\n<p>Also, you could sit down and make predictions by hand and call those external data. That would be completely useless as a ML solution and not useful for the organizer in the future. Try to come up with a definition of what external data should be allowed, and you'll quickly realize how tricky it is. </p>\n\n<p>Now, that doesn't mean that the information provided by Colin is useless, quite the opposite. If you're able to show that the data is useful, maybe there's another way to deduce the data? Some machine learning technique to figure out which towns are close to each other and which ones are far away from each other? Develop a technique using Colins data, then remove Colins data once your technique performs as well without it - and you're in the clear. </p>\n\n<p>If you can generate a winning solution without any external data, then it doesn't matter what data you considered getting there. That's merely the process of feature engineering. </p>\n\n<p>/m</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124887,
      "author_name": "colinpriest",
      "author_url": "",
      "post_date": "06/23/2016 05:47:08",
      "content": "<p>I agree with Matthias.</p>\n\n<p>You can use the latitude and longitude to build a spacial model, then use the information from that to group locations together (maybe by city name or by state). Then your final model uses the grouped locations as features, not the latitudes and longitudes.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124907,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "06/23/2016 09:57:44",
      "content": "<p>Colin,</p>\n\n<p>you will need to explain how you grouped locations together using only the data of the competition.  If you can do so, then great.</p>\n\n<p>I also agree that using external data to better understand the problem is good.   I may look at your map in detail ;)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "124790": "I have written an R script to clean up the location data, geocode and then plot a map. Unfortunately it won't run in Kaggle scripts because the geocoding needs an internet connection.\r\n![enter image description here][1]\r\n\r\n\r\n  [1]: https://s3-ap-southeast-1.amazonaws.com/tutorialfiles/map.png",
    "124824": "I assume that, with this information, you can take the aggregate demand in places that are very close from each other, and model that, instead of modelling each separate store. However, that will only happen when a group of stores are very close... which is when a client will go to a second store if he cannot find his product in the first one.\r\n\r\nI am not sure how large the improvement will be, if any, but may be worth trying.",
    "124852": "HI, beware that you cannot use external data according to the competition rules.  Don't ask me why it is that way...\r\n\r\nI don't see how you can geocode without external data access hence I'm afraid you have to not use this in your submission.",
    "124857": "The rule against external data is fair and not that strange; the competition organizer must (at least potentially) be able to use the solution in production. And if you use external data that may be restricted by licensing issues - or that may degrade over time - that prevents them from using the solutions.\r\n\r\nAlso, you could sit down and make predictions by hand and call those external data. That would be completely useless as a ML solution and not useful for the organizer in the future. Try to come up with a definition of what external data should be allowed, and you'll quickly realize how tricky it is. \r\n\r\nNow, that doesn't mean that the information provided by Colin is useless, quite the opposite. If you're able to show that the data is useful, maybe there's another way to deduce the data? Some machine learning technique to figure out which towns are close to each other and which ones are far away from each other? Develop a technique using Colins data, then remove Colins data once your technique performs as well without it - and you're in the clear. \r\n\r\nIf you can generate a winning solution without any external data, then it doesn't matter what data you considered getting there. That's merely the process of feature engineering. \r\n\r\n/m",
    "124887": "I agree with Matthias.\r\n\r\nYou can use the latitude and longitude to build a spacial model, then use the information from that to group locations together (maybe by city name or by state). Then your final model uses the grouped locations as features, not the latitudes and longitudes.",
    "124907": "Colin,\r\n\r\nyou will need to explain how you grouped locations together using only the data of the competition.  If you can do so, then great.\r\n\r\nI also agree that using external data to better understand the problem is good.   I may look at your map in detail ;)"
  },
  "source": "meta"
}