{
  "id": 41537,
  "title": "External Data Thread",
  "url": "/competitions/favorita-grocery-sales-forecasting/discussion/41537",
  "author_name": "inversion",
  "post_date": "2017-10-19T19:25:15.127000",
  "votes": 23,
  "comment_count": 29,
  "views": 0,
  "content": "<p>Per the <a href=\"https://www.kaggle.com/c/favorita-grocery-sales-forecasting/rules\">competition rules</a>, please post external data in this thread.</p>",
  "messages": [
    {
      "id": 233290,
      "postDate": "2017-10-19T19:25:15.127Z",
      "content": "<p>Per the <a href=\"https://www.kaggle.com/c/favorita-grocery-sales-forecasting/rules\">competition rules</a>, please post external data in this thread.</p>",
      "rawMarkdown": "Per the [competition rules][1], please post external data in this thread.\n\n  [1]: https://www.kaggle.com/c/favorita-grocery-sales-forecasting/rules",
      "votes": 23
    },
    {
      "id": 240658,
      "postDate": "2017-11-07T05:14:00.300Z",
      "content": "<p>Detailed weather data, using dark sky API in R:\n<a href=\"https://darksky.net/\">https://darksky.net/</a></p>\n\n<p>I know some have expressed concern over using weather data, and it being impractical. Weather forecasts are actually routinely used in retailer / consumer products forecasting models. The absolute temps are not relevant, but significant deviations from historical norms are.</p>",
      "rawMarkdown": "Detailed weather data, using dark sky API in R:\nhttps://darksky.net/\n\nI know some have expressed concern over using weather data, and it being impractical. Weather forecasts are actually routinely used in retailer / consumer products forecasting models. The absolute temps are not relevant, but significant deviations from historical norms are.",
      "votes": 9
    },
    {
      "id": 240225,
      "postDate": "2017-11-06T00:56:16.970Z",
      "content": "<p>Consumer price index (inflation) in Ecuador (between 2015-01 and 2017-09):</p>\n\n<p><a href=\"http://www.ecuadorencifras.gob.ec/documentos/web-inec/Inflacion/2017/Septiembre-2017/DECON_IPC_CSV_2017_09.zip\">http://www.ecuadorencifras.gob.ec/documentos/web-inec/Inflacion/2017/Septiembre-2017/DECON_IPC_CSV_2017_09.zip</a></p>",
      "rawMarkdown": "Consumer price index (inflation) in Ecuador (between 2015-01 and 2017-09):\n\nhttp://www.ecuadorencifras.gob.ec/documentos/web-inec/Inflacion/2017/Septiembre-2017/DECON_IPC_CSV_2017_09.zip",
      "votes": 7
    },
    {
      "id": 248196,
      "postDate": "2017-11-25T07:44:00.527Z",
      "content": "<p>Weather data \n<a href=\"https://www.ncdc.noaa.gov/cdo-web/datasets\">https://www.ncdc.noaa.gov/cdo-web/datasets</a></p>",
      "rawMarkdown": "Weather data \n[https://www.ncdc.noaa.gov/cdo-web/datasets][1]\n  [1]: https://www.ncdc.noaa.gov/cdo-web/datasets",
      "votes": 6,
      "replies": [
        {
          "id": 252823,
          "postDate": "2017-12-03T23:56:32.890Z",
          "content": "<p>Annoying that the Ecuador data has so many holes, but at least it's free data!  I can't find anything to download from Ecuador's weather service though...</p>",
          "rawMarkdown": "Annoying that the Ecuador data has so many holes, but at least it's free data!  I can't find anything to download from Ecuador's weather service though..."
        }
      ]
    },
    {
      "id": 240226,
      "postDate": "2017-11-06T01:02:27.093Z",
      "content": "<p>National Institute of Statistics of Ecuador: </p>\n\n<p><a href=\"http://www.ecuadorencifras.gob.ec/estadisticas/\">http://www.ecuadorencifras.gob.ec/estadisticas/</a></p>\n\n<p>Contains many historical social and economic statistics of Ecuador.</p>",
      "rawMarkdown": "National Institute of Statistics of Ecuador: \n\nhttp://www.ecuadorencifras.gob.ec/estadisticas/\n\nContains many historical social and economic statistics of Ecuador.\n",
      "votes": 3
    },
    {
      "id": 240221,
      "postDate": "2017-11-06T00:32:12.583Z",
      "content": "<p>Ecuador Equity Market Stats:\n<a href=\"http://www.bolsadequito.info/estadisticas/informacion-estadistica/\">http://www.bolsadequito.info/estadisticas/informacion-estadistica/</a></p>",
      "rawMarkdown": "Ecuador Equity Market Stats:\nhttp://www.bolsadequito.info/estadisticas/informacion-estadistica/\n",
      "votes": 3
    },
    {
      "id": 234737,
      "postDate": "2017-10-23T23:11:24.980Z",
      "content": "<p>Hello, I'm charging Census for the cities, from the link :</p>\n\n<p><a href=\"http://redatam.inec.gob.ec/cgibin/RpWebEngine.exe/PortalAction?&amp;MODE=MAIN&amp;BASE=CPV2010&amp;MAIN=WebServerMain.inl\">http://redatam.inec.gob.ec/cgibin/RpWebEngine.exe/PortalAction?&amp;MODE=MAIN&amp;BASE=CPV2010&amp;MAIN=WebServerMain.inl</a></p>",
      "rawMarkdown": "Hello, I'm charging Census for the cities, from the link :\n\nhttp://redatam.inec.gob.ec/cgibin/RpWebEngine.exe/PortalAction?&amp;MODE=MAIN&amp;BASE=CPV2010&amp;MAIN=WebServerMain.inl\n",
      "votes": 3
    },
    {
      "id": 265077,
      "postDate": "2018-01-04T14:54:30.293Z",
      "content": "<p>Weather data from <a href=\"https://darksky.net/\">https://darksky.net</a> using their API (for country / city etc) + coordinates from google maps</p>",
      "rawMarkdown": "Weather data from https://darksky.net using their API (for country / city etc) + coordinates from google maps",
      "votes": 1
    },
    {
      "id": 247922,
      "postDate": "2017-11-24T10:09:51.553Z",
      "content": "<p>I know many of this are already given, but here a small recap of some sources:</p>\n\n<p><strong>Data from the World Bank</strong>\n<a href=\"https://data.worldbank.org/country/ecuador\">https://data.worldbank.org/country/ecuador</a> \ndocument: <a href=\"http://api.worldbank.org/v2/en/country/ECU?downloadformat=csv\">http://api.worldbank.org/v2/en/country/ECU?downloadformat=csv</a></p>\n\n<p>There is some microdata available but I haven't registered yet to get it\n<a href=\"https://microdata.worldbank.org/index.php/auth/login/?destination=catalog/2413/get_microdata\">https://microdata.worldbank.org/index.php/auth/login/?destination=catalog/2413/get_microdata</a></p>\n\n<p><strong>There is some data in Datahub</strong>\n<a href=\"https://old.datahub.io/dataset/opendataec\">https://old.datahub.io/dataset/opendataec</a>\ndocument: <a href=\"https://old.datahub.io/dataset/opendataec/datapackage.json\">https://old.datahub.io/dataset/opendataec/datapackage.json</a></p>\n\n<p><strong>Instituto nacional de estadísticas y sensos del ecuador</strong>\n<a href=\"http://www.ecuadorencifras.gob.ec/institucional/home/\">http://www.ecuadorencifras.gob.ec/institucional/home/</a></p>\n\n<p><strong>Datos de CESLA Cento de estudios latinoamericanos</strong></p>\n\n<p><a href=\"https://www.cesla.com/estadisticas-economia-ecuador.php\">https://www.cesla.com/estadisticas-economia-ecuador.php</a> (also needs registration, I haven't done that yet)</p>\n\n<p><strong>Banco central del ecuador:</strong>\n<a href=\"https://www.bce.fin.ec/index.php/indicadores-economicos\">https://www.bce.fin.ec/index.php/indicadores-economicos</a></p>\n\n<p><a href=\"https://www.bce.fin.ec/index.php/component/k2/item/776\">https://www.bce.fin.ec/index.php/component/k2/item/776</a>  (also needs registration, I haven't done that yet)</p>",
      "rawMarkdown": "I know many of this are already given, but here a small recap of some sources:\n\n**Data from the World Bank**\nhttps://data.worldbank.org/country/ecuador \ndocument: http://api.worldbank.org/v2/en/country/ECU?downloadformat=csv\n\nThere is some microdata available but I haven't registered yet to get it\nhttps://microdata.worldbank.org/index.php/auth/login/?destination=catalog/2413/get_microdata\n\n**There is some data in Datahub**\nhttps://old.datahub.io/dataset/opendataec\ndocument: https://old.datahub.io/dataset/opendataec/datapackage.json\n\n**Instituto nacional de estadísticas y sensos del ecuador**\nhttp://www.ecuadorencifras.gob.ec/institucional/home/\n\n\n**Datos de CESLA Cento de estudios latinoamericanos**\n\nhttps://www.cesla.com/estadisticas-economia-ecuador.php (also needs registration, I haven't done that yet)\n\n\n**Banco central del ecuador:**\nhttps://www.bce.fin.ec/index.php/indicadores-economicos\n\nhttps://www.bce.fin.ec/index.php/component/k2/item/776  (also needs registration, I haven't done that yet)\n\n",
      "votes": 1
    },
    {
      "id": 240076,
      "postDate": "2017-11-05T14:58:57.157Z",
      "content": "<p>@inversion, Is using of weather data permitted (given that those are posted here)? I mean, weather does influence sales of certain products, but in real live situation, it's challenging to make  an accurate forecast for 2 weeks (test period), so including weather data in feature set is kind of a cheating.  On the other hand, posted data include oil prices covering test period, so I guess the sponsors don't mind data leakage. Which is cool, as long as they realize that models using these kind of 'future telling' features might be less accurate in real life setting.</p>",
      "rawMarkdown": "@inversion, Is using of weather data permitted (given that those are posted here)? I mean, weather does influence sales of certain products, but in real live situation, it's challenging to make  an accurate forecast for 2 weeks (test period), so including weather data in feature set is kind of a cheating.  On the other hand, posted data include oil prices covering test period, so I guess the sponsors don't mind data leakage. Which is cool, as long as they realize that models using these kind of 'future telling' features might be less accurate in real life setting.",
      "votes": 1,
      "replies": [
        {
          "id": 240584,
          "postDate": "2017-11-06T23:04:03.663Z",
          "content": "<p>Hi Andrey -</p>\n\n<p>Yes, even though it's not quite \"natural\" to have complete weather data for the test period, you can use it for this competition, as long as the information is posted as per the competition rules</p>",
          "rawMarkdown": "Hi Andrey -\n\nYes, even though it's not quite \"natural\" to have complete weather data for the test period, you can use it for this competition, as long as the information is posted as per the competition rules",
          "votes": 1
        }
      ]
    },
    {
      "id": 237790,
      "postDate": "2017-10-31T00:36:06.167Z",
      "content": "<p>Global Price of Cocoa from this link <a href=\"https://fred.stlouisfed.org/series/PCOCOUSDM\">https://fred.stlouisfed.org/series/PCOCOUSDM</a></p>\n\n<p>Global Price of Bananas from this link  <a href=\"https://fred.stlouisfed.org/series/PBANSOPUSDM\">https://fred.stlouisfed.org/series/PBANSOPUSDM</a></p>\n\n<p>Population of city from this link  <a href=\"http://data.un.org/Data.aspx?d=POP&amp;f=tableCode%3A240\">http://data.un.org/Data.aspx?d=POP&amp;f=tableCode%3A240</a></p>",
      "rawMarkdown": "Global Price of Cocoa from this link https://fred.stlouisfed.org/series/PCOCOUSDM\n\nGlobal Price of Bananas from this link  https://fred.stlouisfed.org/series/PBANSOPUSDM\n\nPopulation of city from this link  http://data.un.org/Data.aspx?d=POP&amp;f=tableCode%3A240",
      "votes": 1
    },
    {
      "id": 234740,
      "postDate": "2017-10-23T23:17:09.167Z",
      "content": "<p>Population Proyections\nI found this information from this link:</p>\n\n<p><a href=\"http://www.ecuadorencifras.gob.ec/vdatos/\">http://www.ecuadorencifras.gob.ec/vdatos/</a></p>",
      "rawMarkdown": "Population Proyections\nI found this information from this link:\n\nhttp://www.ecuadorencifras.gob.ec/vdatos/\n",
      "votes": 1
    },
    {
      "id": 261139,
      "postDate": "2017-12-21T21:53:01.317Z",
      "content": "<p>More information about Holidays in Ecuador, from a local site:</p>\n\n<p><a href=\"http://www.feriadosecuador.net/feriado/2014\">http://www.feriadosecuador.net/feriado/2014</a></p>\n\n<p><a href=\"http://www.feriadosecuador.net/feriado/2015\">http://www.feriadosecuador.net/feriado/2015</a></p>\n\n<p><a href=\"http://www.feriadosecuador.net/feriado/2016\">http://www.feriadosecuador.net/feriado/2016</a></p>\n\n<p><a href=\"http://www.feriadosecuador.net/feriado/2017\">http://www.feriadosecuador.net/feriado/2017</a></p>",
      "rawMarkdown": "More information about Holidays in Ecuador, from a local site:\n\nhttp://www.feriadosecuador.net/feriado/2014\n\nhttp://www.feriadosecuador.net/feriado/2015\n\nhttp://www.feriadosecuador.net/feriado/2016\n\nhttp://www.feriadosecuador.net/feriado/2017\n\n",
      "votes": 2
    },
    {
      "id": 255991,
      "postDate": "2017-12-10T21:32:43.773Z",
      "content": "<p>google trends <a href=\"https://trends.google.com/trends/explore?date=today%205-y&amp;q=Corporaci%C3%B3n%20Favorita\">https://trends.google.com/trends/explore?date=today%205-y&amp;q=Corporaci%C3%B3n%20Favorita</a></p>",
      "rawMarkdown": "google trends https://trends.google.com/trends/explore?date=today%205-y&amp;q=Corporaci%C3%B3n%20Favorita",
      "votes": 2
    },
    {
      "id": 234104,
      "postDate": "2017-10-22T10:13:47.720Z",
      "content": "<p>Census data for the cities (if available).</p>",
      "rawMarkdown": "Census data for the cities (if available).",
      "votes": 2
    },
    {
      "id": 266559,
      "postDate": "2018-01-09T04:19:39.240Z",
      "content": "<p>City &lt;-&gt; State (Provincia) mapping</p>\n\n<p><a href=\"https://es.wikipedia.org/wiki/Anexo:Ciudades_de_Ecuador\">https://es.wikipedia.org/wiki/Anexo:Ciudades_de_Ecuador</a></p>",
      "rawMarkdown": "City &lt;-&gt; State (Provincia) mapping\n\nhttps://es.wikipedia.org/wiki/Anexo:Ciudades_de_Ecuador"
    },
    {
      "id": 266166,
      "postDate": "2018-01-08T01:27:58.190Z",
      "content": "<p><a href=\"https://en.wikipedia.org/wiki/2018_FIFA_World_Cup_qualification_(CONMEBOL)\">https://en.wikipedia.org/wiki/2018_FIFA_World_Cup_qualification_(CONMEBOL)</a></p>",
      "rawMarkdown": "https://en.wikipedia.org/wiki/2018_FIFA_World_Cup_qualification_(CONMEBOL)"
    },
    {
      "id": 265965,
      "postDate": "2018-01-07T09:10:52.140Z",
      "content": "<p>not sure if it counts as external data, but I better list it - manual mapping from stores to regions (Costa, Sierra, Oriente) (wasn't able to find it as data so have to do myself)</p>",
      "rawMarkdown": "not sure if it counts as external data, but I better list it - manual mapping from stores to regions (Costa, Sierra, Oriente) (wasn't able to find it as data so have to do myself)"
    },
    {
      "id": 253226,
      "postDate": "2017-12-04T16:46:47.280Z",
      "content": "<p>Could you clarify the rules with regards to external sources please. According to the rules: \"The source of any external data must be posted to the official competition forum prior to the First Submission Deadline.\" Does that mean in this case the deadline for posting external sources is January 15, 2018 @ 11:59 PM UTC?</p>",
      "rawMarkdown": "Could you clarify the rules with regards to external sources please. According to the rules: \"The source of any external data must be posted to the official competition forum prior to the First Submission Deadline.\" Does that mean in this case the deadline for posting external sources is January 15, 2018 @ 11:59 PM UTC?"
    },
    {
      "id": 252802,
      "postDate": "2017-12-03T22:39:52.540Z",
      "content": "<p>This is potentially a really dumb question, but if I'm using an external library like scipy, do I need to post that here? Would that count as \"external data\" under the definition given in the rules?</p>\n\n<p>This part of the rules (below) confused me a bit, hence the question.</p>\n\n<p><em>\"Use of publicly available external data is permitted, provided it does not pertain to Corporación Favorita nor to any entity in the same line of business as Corporación Favorita. <strong>For example, you may use pre-trained networks or natural language tools/libraries</strong>.\"</em></p>\n\n<p>(Just to be on the safe side, pending an answer, I'm using scikit-learn, scipy, and numpy.)</p>",
      "rawMarkdown": "This is potentially a really dumb question, but if I'm using an external library like scipy, do I need to post that here? Would that count as \"external data\" under the definition given in the rules?\n\nThis part of the rules (below) confused me a bit, hence the question.\n\n*\"Use of publicly available external data is permitted, provided it does not pertain to Corporación Favorita nor to any entity in the same line of business as Corporación Favorita. **For example, you may use pre-trained networks or natural language tools/libraries**.\"*\n\n(Just to be on the safe side, pending an answer, I'm using scikit-learn, scipy, and numpy.)",
      "replies": [
        {
          "id": 252821,
          "postDate": "2017-12-03T23:53:45.663Z",
          "content": "<p>They fall under properly licensed open source code - and all three of those are in the Kaggle Docker images and hence kernel engine.  </p>\n\n<p>I'd say anything you can directly access in kernels is fair game, unless you find something that is improperly licensed.</p>",
          "rawMarkdown": "They fall under properly licensed open source code - and all three of those are in the Kaggle Docker images and hence kernel engine.  \n\nI'd say anything you can directly access in kernels is fair game, unless you find something that is improperly licensed."
        }
      ]
    },
    {
      "id": 246545,
      "postDate": "2017-11-21T09:09:58.730Z",
      "content": "<p>is there any way to get product descriptions or titles?</p>",
      "rawMarkdown": "is there any way to get product descriptions or titles?",
      "replies": [
        {
          "id": 247355,
          "postDate": "2017-11-22T21:38:06.277Z",
          "content": "<p>Here maybe. Not sure if that is useful or allowed.</p>\n\n<p><a href=\"https://www.cfavorita.ec/catalogo/pages/articulos/reporteArticulos.jsf?stb=FMfNi9lSmJU=&amp;stb=SKhwvCs8Vjk=&amp;stb=d0DdL5B0tMQ=&amp;stb=lnglDrFCCbo=&amp;stb=ngGiGCP1QUo=\">https://www.cfavorita.ec/catalogo/pages/articulos/reporteArticulos.jsf?stb=FMfNi9lSmJU=&amp;stb=SKhwvCs8Vjk=&amp;stb=d0DdL5B0tMQ=&amp;stb=lnglDrFCCbo=&amp;stb=ngGiGCP1QUo=</a></p>",
          "rawMarkdown": "Here maybe. Not sure if that is useful or allowed.\n\nhttps://www.cfavorita.ec/catalogo/pages/articulos/reporteArticulos.jsf?stb=FMfNi9lSmJU=&amp;stb=SKhwvCs8Vjk=&amp;stb=d0DdL5B0tMQ=&amp;stb=lnglDrFCCbo=&amp;stb=ngGiGCP1QUo=",
          "votes": 1
        }
      ]
    },
    {
      "id": 240517,
      "postDate": "2017-11-06T19:50:41.373Z",
      "content": "<p>There is a bunch of stuff in <a href=\"https://data.worldbank.org/country/ecuador?view=chart\">worldbank</a> - mostly quarterly data, on country level. Note that is is posted usually with a few months delay, so it's best to shift the data 3 months.</p>",
      "rawMarkdown": "There is a bunch of stuff in [worldbank][1] - mostly quarterly data, on country level. Note that is is posted usually with a few months delay, so it's best to shift the data 3 months.\n\n\n  [1]: https://data.worldbank.org/country/ecuador?view=chart"
    },
    {
      "id": 238992,
      "postDate": "2017-11-02T13:52:50.670Z",
      "content": "<p>\" you may not use data other than the Competition Data to develop and test your models and Submissions\"  -- Rule 7C.</p>\n\n<p>What is the purpose of having this External Data Thread, when the rule above exists?</p>",
      "rawMarkdown": "\" you may not use data other than the Competition Data to develop and test your models and Submissions\"  -- Rule 7C.\n\nWhat is the purpose of having this External Data Thread, when the rule above exists?",
      "replies": [
        {
          "id": 238999,
          "postDate": "2017-11-02T14:13:57.677Z",
          "content": "<p>\"<strong>Unless otherwise expressly stated on the Competition Website</strong>, you may not use data other than the Competition Data to develop and test your models and Submissions.\"</p>\n\n<p>Above that, it reads \n\"EXTERNAL DATA: Permitted, with guidelines as follows,</p>\n\n<pre><code>Use of publicly available external data is permitted, provided it does not pertain to Corporación Favorita nor to any entity in the same line of business as Corporación Favorita. For example, you may use pre-trained networks or natural language tools/libraries.\n\nThe source of any external data must be posted to the official competition forum prior to the First Submission Deadline.\"\n</code></pre>",
          "rawMarkdown": "\"**Unless otherwise expressly stated on the Competition Website**, you may not use data other than the Competition Data to develop and test your models and Submissions.\"\n\nAbove that, it reads \n\"EXTERNAL DATA: Permitted, with guidelines as follows,\n\n    Use of publicly available external data is permitted, provided it does not pertain to Corporación Favorita nor to any entity in the same line of business as Corporación Favorita. For example, you may use pre-trained networks or natural language tools/libraries.\n\n    The source of any external data must be posted to the official competition forum prior to the First Submission Deadline.\"\n",
          "votes": 1
        }
      ]
    },
    {
      "id": 249821,
      "postDate": "2017-11-29T09:12:18.027Z",
      "rawMarkdown": "",
      "votes": 2,
      "isDeleted": true
    },
    {
      "id": 237239,
      "postDate": "2017-10-29T22:49:29.660Z",
      "rawMarkdown": "",
      "votes": 5,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 240658,
      "author_name": "ArminKakas",
      "author_url": "",
      "post_date": "2017-11-07T05:14:00.300000",
      "content": "<p>Detailed weather data, using dark sky API in R:\n<a href=\"https://darksky.net/\">https://darksky.net/</a></p>\n\n<p>I know some have expressed concern over using weather data, and it being impractical. Weather forecasts are actually routinely used in retailer / consumer products forecasting models. The absolute temps are not relevant, but significant deviations from historical norms are.</p>",
      "votes": 9,
      "replies": []
    },
    {
      "id": 240225,
      "author_name": "Paulo Pinto",
      "author_url": "",
      "post_date": "2017-11-06T00:56:16.970000",
      "content": "<p>Consumer price index (inflation) in Ecuador (between 2015-01 and 2017-09):</p>\n\n<p><a href=\"http://www.ecuadorencifras.gob.ec/documentos/web-inec/Inflacion/2017/Septiembre-2017/DECON_IPC_CSV_2017_09.zip\">http://www.ecuadorencifras.gob.ec/documentos/web-inec/Inflacion/2017/Septiembre-2017/DECON_IPC_CSV_2017_09.zip</a></p>",
      "votes": 7,
      "replies": []
    },
    {
      "id": 248196,
      "author_name": "Belinda Trotta",
      "author_url": "",
      "post_date": "2017-11-25T07:44:00.527000",
      "content": "<p>Weather data \n<a href=\"https://www.ncdc.noaa.gov/cdo-web/datasets\">https://www.ncdc.noaa.gov/cdo-web/datasets</a></p>",
      "votes": 6,
      "replies": [
        {
          "id": 252823,
          "author_name": "happycube",
          "author_url": "",
          "post_date": "2017-12-03T23:56:32.890000",
          "content": "<p>Annoying that the Ecuador data has so many holes, but at least it's free data!  I can't find anything to download from Ecuador's weather service though...</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 240226,
      "author_name": "Paulo Pinto",
      "author_url": "",
      "post_date": "2017-11-06T01:02:27.093000",
      "content": "<p>National Institute of Statistics of Ecuador: </p>\n\n<p><a href=\"http://www.ecuadorencifras.gob.ec/estadisticas/\">http://www.ecuadorencifras.gob.ec/estadisticas/</a></p>\n\n<p>Contains many historical social and economic statistics of Ecuador.</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 240221,
      "author_name": "Paulo Pinto",
      "author_url": "",
      "post_date": "2017-11-06T00:32:12.583000",
      "content": "<p>Ecuador Equity Market Stats:\n<a href=\"http://www.bolsadequito.info/estadisticas/informacion-estadistica/\">http://www.bolsadequito.info/estadisticas/informacion-estadistica/</a></p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 234737,
      "author_name": "Elio",
      "author_url": "",
      "post_date": "2017-10-23T23:11:24.980000",
      "content": "<p>Hello, I'm charging Census for the cities, from the link :</p>\n\n<p><a href=\"http://redatam.inec.gob.ec/cgibin/RpWebEngine.exe/PortalAction?&amp;MODE=MAIN&amp;BASE=CPV2010&amp;MAIN=WebServerMain.inl\">http://redatam.inec.gob.ec/cgibin/RpWebEngine.exe/PortalAction?&amp;MODE=MAIN&amp;BASE=CPV2010&amp;MAIN=WebServerMain.inl</a></p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 265077,
      "author_name": "Radek Osmulski",
      "author_url": "",
      "post_date": "2018-01-04T14:54:30.293000",
      "content": "<p>Weather data from <a href=\"https://darksky.net/\">https://darksky.net</a> using their API (for country / city etc) + coordinates from google maps</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 247922,
      "author_name": "lamorocha",
      "author_url": "",
      "post_date": "2017-11-24T10:09:51.553000",
      "content": "<p>I know many of this are already given, but here a small recap of some sources:</p>\n\n<p><strong>Data from the World Bank</strong>\n<a href=\"https://data.worldbank.org/country/ecuador\">https://data.worldbank.org/country/ecuador</a> \ndocument: <a href=\"http://api.worldbank.org/v2/en/country/ECU?downloadformat=csv\">http://api.worldbank.org/v2/en/country/ECU?downloadformat=csv</a></p>\n\n<p>There is some microdata available but I haven't registered yet to get it\n<a href=\"https://microdata.worldbank.org/index.php/auth/login/?destination=catalog/2413/get_microdata\">https://microdata.worldbank.org/index.php/auth/login/?destination=catalog/2413/get_microdata</a></p>\n\n<p><strong>There is some data in Datahub</strong>\n<a href=\"https://old.datahub.io/dataset/opendataec\">https://old.datahub.io/dataset/opendataec</a>\ndocument: <a href=\"https://old.datahub.io/dataset/opendataec/datapackage.json\">https://old.datahub.io/dataset/opendataec/datapackage.json</a></p>\n\n<p><strong>Instituto nacional de estadísticas y sensos del ecuador</strong>\n<a href=\"http://www.ecuadorencifras.gob.ec/institucional/home/\">http://www.ecuadorencifras.gob.ec/institucional/home/</a></p>\n\n<p><strong>Datos de CESLA Cento de estudios latinoamericanos</strong></p>\n\n<p><a href=\"https://www.cesla.com/estadisticas-economia-ecuador.php\">https://www.cesla.com/estadisticas-economia-ecuador.php</a> (also needs registration, I haven't done that yet)</p>\n\n<p><strong>Banco central del ecuador:</strong>\n<a href=\"https://www.bce.fin.ec/index.php/indicadores-economicos\">https://www.bce.fin.ec/index.php/indicadores-economicos</a></p>\n\n<p><a href=\"https://www.bce.fin.ec/index.php/component/k2/item/776\">https://www.bce.fin.ec/index.php/component/k2/item/776</a>  (also needs registration, I haven't done that yet)</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 240076,
      "author_name": "Andrey",
      "author_url": "",
      "post_date": "2017-11-05T14:58:57.157000",
      "content": "<p>@inversion, Is using of weather data permitted (given that those are posted here)? I mean, weather does influence sales of certain products, but in real live situation, it's challenging to make  an accurate forecast for 2 weeks (test period), so including weather data in feature set is kind of a cheating.  On the other hand, posted data include oil prices covering test period, so I guess the sponsors don't mind data leakage. Which is cool, as long as they realize that models using these kind of 'future telling' features might be less accurate in real life setting.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 240584,
          "author_name": "inversion",
          "author_url": "",
          "post_date": "2017-11-06T23:04:03.663000",
          "content": "<p>Hi Andrey -</p>\n\n<p>Yes, even though it's not quite \"natural\" to have complete weather data for the test period, you can use it for this competition, as long as the information is posted as per the competition rules</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 237790,
      "author_name": "Frankie Zeager",
      "author_url": "",
      "post_date": "2017-10-31T00:36:06.167000",
      "content": "<p>Global Price of Cocoa from this link <a href=\"https://fred.stlouisfed.org/series/PCOCOUSDM\">https://fred.stlouisfed.org/series/PCOCOUSDM</a></p>\n\n<p>Global Price of Bananas from this link  <a href=\"https://fred.stlouisfed.org/series/PBANSOPUSDM\">https://fred.stlouisfed.org/series/PBANSOPUSDM</a></p>\n\n<p>Population of city from this link  <a href=\"http://data.un.org/Data.aspx?d=POP&amp;f=tableCode%3A240\">http://data.un.org/Data.aspx?d=POP&amp;f=tableCode%3A240</a></p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 234740,
      "author_name": "Elio",
      "author_url": "",
      "post_date": "2017-10-23T23:17:09.167000",
      "content": "<p>Population Proyections\nI found this information from this link:</p>\n\n<p><a href=\"http://www.ecuadorencifras.gob.ec/vdatos/\">http://www.ecuadorencifras.gob.ec/vdatos/</a></p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 261139,
      "author_name": "Kenneth B",
      "author_url": "",
      "post_date": "2017-12-21T21:53:01.317000",
      "content": "<p>More information about Holidays in Ecuador, from a local site:</p>\n\n<p><a href=\"http://www.feriadosecuador.net/feriado/2014\">http://www.feriadosecuador.net/feriado/2014</a></p>\n\n<p><a href=\"http://www.feriadosecuador.net/feriado/2015\">http://www.feriadosecuador.net/feriado/2015</a></p>\n\n<p><a href=\"http://www.feriadosecuador.net/feriado/2016\">http://www.feriadosecuador.net/feriado/2016</a></p>\n\n<p><a href=\"http://www.feriadosecuador.net/feriado/2017\">http://www.feriadosecuador.net/feriado/2017</a></p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 255991,
      "author_name": "Glib Kechyn",
      "author_url": "",
      "post_date": "2017-12-10T21:32:43.773000",
      "content": "<p>google trends <a href=\"https://trends.google.com/trends/explore?date=today%205-y&amp;q=Corporaci%C3%B3n%20Favorita\">https://trends.google.com/trends/explore?date=today%205-y&amp;q=Corporaci%C3%B3n%20Favorita</a></p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 234104,
      "author_name": "Dan Ofer",
      "author_url": "",
      "post_date": "2017-10-22T10:13:47.720000",
      "content": "<p>Census data for the cities (if available).</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 266559,
      "author_name": "Romeo Cabrera",
      "author_url": "",
      "post_date": "2018-01-09T04:19:39.240000",
      "content": "<p>City &lt;-&gt; State (Provincia) mapping</p>\n\n<p><a href=\"https://es.wikipedia.org/wiki/Anexo:Ciudades_de_Ecuador\">https://es.wikipedia.org/wiki/Anexo:Ciudades_de_Ecuador</a></p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 266166,
      "author_name": "Romeo Cabrera",
      "author_url": "",
      "post_date": "2018-01-08T01:27:58.190000",
      "content": "<p><a href=\"https://en.wikipedia.org/wiki/2018_FIFA_World_Cup_qualification_(CONMEBOL)\">https://en.wikipedia.org/wiki/2018_FIFA_World_Cup_qualification_(CONMEBOL)</a></p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 265965,
      "author_name": "steelrose",
      "author_url": "",
      "post_date": "2018-01-07T09:10:52.140000",
      "content": "<p>not sure if it counts as external data, but I better list it - manual mapping from stores to regions (Costa, Sierra, Oriente) (wasn't able to find it as data so have to do myself)</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 253226,
      "author_name": "mikola",
      "author_url": "",
      "post_date": "2017-12-04T16:46:47.280000",
      "content": "<p>Could you clarify the rules with regards to external sources please. According to the rules: \"The source of any external data must be posted to the official competition forum prior to the First Submission Deadline.\" Does that mean in this case the deadline for posting external sources is January 15, 2018 @ 11:59 PM UTC?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 252802,
      "author_name": "Christopher Hicham Moran",
      "author_url": "",
      "post_date": "2017-12-03T22:39:52.540000",
      "content": "<p>This is potentially a really dumb question, but if I'm using an external library like scipy, do I need to post that here? Would that count as \"external data\" under the definition given in the rules?</p>\n\n<p>This part of the rules (below) confused me a bit, hence the question.</p>\n\n<p><em>\"Use of publicly available external data is permitted, provided it does not pertain to Corporación Favorita nor to any entity in the same line of business as Corporación Favorita. <strong>For example, you may use pre-trained networks or natural language tools/libraries</strong>.\"</em></p>\n\n<p>(Just to be on the safe side, pending an answer, I'm using scikit-learn, scipy, and numpy.)</p>",
      "votes": 0,
      "replies": [
        {
          "id": 252821,
          "author_name": "happycube",
          "author_url": "",
          "post_date": "2017-12-03T23:53:45.663000",
          "content": "<p>They fall under properly licensed open source code - and all three of those are in the Kaggle Docker images and hence kernel engine.  </p>\n\n<p>I'd say anything you can directly access in kernels is fair game, unless you find something that is improperly licensed.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 246545,
      "author_name": "jony rosenski",
      "author_url": "",
      "post_date": "2017-11-21T09:09:58.730000",
      "content": "<p>is there any way to get product descriptions or titles?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 247355,
          "author_name": "Batangas",
          "author_url": "",
          "post_date": "2017-11-22T21:38:06.277000",
          "content": "<p>Here maybe. Not sure if that is useful or allowed.</p>\n\n<p><a href=\"https://www.cfavorita.ec/catalogo/pages/articulos/reporteArticulos.jsf?stb=FMfNi9lSmJU=&amp;stb=SKhwvCs8Vjk=&amp;stb=d0DdL5B0tMQ=&amp;stb=lnglDrFCCbo=&amp;stb=ngGiGCP1QUo=\">https://www.cfavorita.ec/catalogo/pages/articulos/reporteArticulos.jsf?stb=FMfNi9lSmJU=&amp;stb=SKhwvCs8Vjk=&amp;stb=d0DdL5B0tMQ=&amp;stb=lnglDrFCCbo=&amp;stb=ngGiGCP1QUo=</a></p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 240517,
      "author_name": "hristo buyukliev",
      "author_url": "",
      "post_date": "2017-11-06T19:50:41.373000",
      "content": "<p>There is a bunch of stuff in <a href=\"https://data.worldbank.org/country/ecuador?view=chart\">worldbank</a> - mostly quarterly data, on country level. Note that is is posted usually with a few months delay, so it's best to shift the data 3 months.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 238992,
      "author_name": "Geoffrey Anderson",
      "author_url": "",
      "post_date": "2017-11-02T13:52:50.670000",
      "content": "<p>\" you may not use data other than the Competition Data to develop and test your models and Submissions\"  -- Rule 7C.</p>\n\n<p>What is the purpose of having this External Data Thread, when the rule above exists?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 238999,
          "author_name": "Frankie Zeager",
          "author_url": "",
          "post_date": "2017-11-02T14:13:57.677000",
          "content": "<p>\"<strong>Unless otherwise expressly stated on the Competition Website</strong>, you may not use data other than the Competition Data to develop and test your models and Submissions.\"</p>\n\n<p>Above that, it reads \n\"EXTERNAL DATA: Permitted, with guidelines as follows,</p>\n\n<pre><code>Use of publicly available external data is permitted, provided it does not pertain to Corporación Favorita nor to any entity in the same line of business as Corporación Favorita. For example, you may use pre-trained networks or natural language tools/libraries.\n\nThe source of any external data must be posted to the official competition forum prior to the First Submission Deadline.\"\n</code></pre>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 249821,
      "author_name": "",
      "author_url": "",
      "post_date": "2017-11-29T09:12:18.027000",
      "content": "",
      "votes": 2,
      "replies": []
    },
    {
      "id": 237239,
      "author_name": "",
      "author_url": "",
      "post_date": "2017-10-29T22:49:29.660000",
      "content": "",
      "votes": 5,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "233290": "Per the [competition rules][1], please post external data in this thread.\n\n  [1]: https://www.kaggle.com/c/favorita-grocery-sales-forecasting/rules",
    "240658": "Detailed weather data, using dark sky API in R:\nhttps://darksky.net/\n\nI know some have expressed concern over using weather data, and it being impractical. Weather forecasts are actually routinely used in retailer / consumer products forecasting models. The absolute temps are not relevant, but significant deviations from historical norms are.",
    "240225": "Consumer price index (inflation) in Ecuador (between 2015-01 and 2017-09):\n\nhttp://www.ecuadorencifras.gob.ec/documentos/web-inec/Inflacion/2017/Septiembre-2017/DECON_IPC_CSV_2017_09.zip",
    "248196": "Weather data \n[https://www.ncdc.noaa.gov/cdo-web/datasets][1]\n  [1]: https://www.ncdc.noaa.gov/cdo-web/datasets",
    "240226": "National Institute of Statistics of Ecuador: \n\nhttp://www.ecuadorencifras.gob.ec/estadisticas/\n\nContains many historical social and economic statistics of Ecuador.\n",
    "240221": "Ecuador Equity Market Stats:\nhttp://www.bolsadequito.info/estadisticas/informacion-estadistica/\n",
    "234737": "Hello, I'm charging Census for the cities, from the link :\n\nhttp://redatam.inec.gob.ec/cgibin/RpWebEngine.exe/PortalAction?&amp;MODE=MAIN&amp;BASE=CPV2010&amp;MAIN=WebServerMain.inl\n",
    "265077": "Weather data from https://darksky.net using their API (for country / city etc) + coordinates from google maps",
    "247922": "I know many of this are already given, but here a small recap of some sources:\n\n**Data from the World Bank**\nhttps://data.worldbank.org/country/ecuador \ndocument: http://api.worldbank.org/v2/en/country/ECU?downloadformat=csv\n\nThere is some microdata available but I haven't registered yet to get it\nhttps://microdata.worldbank.org/index.php/auth/login/?destination=catalog/2413/get_microdata\n\n**There is some data in Datahub**\nhttps://old.datahub.io/dataset/opendataec\ndocument: https://old.datahub.io/dataset/opendataec/datapackage.json\n\n**Instituto nacional de estadísticas y sensos del ecuador**\nhttp://www.ecuadorencifras.gob.ec/institucional/home/\n\n\n**Datos de CESLA Cento de estudios latinoamericanos**\n\nhttps://www.cesla.com/estadisticas-economia-ecuador.php (also needs registration, I haven't done that yet)\n\n\n**Banco central del ecuador:**\nhttps://www.bce.fin.ec/index.php/indicadores-economicos\n\nhttps://www.bce.fin.ec/index.php/component/k2/item/776  (also needs registration, I haven't done that yet)\n\n",
    "240076": "@inversion, Is using of weather data permitted (given that those are posted here)? I mean, weather does influence sales of certain products, but in real live situation, it's challenging to make  an accurate forecast for 2 weeks (test period), so including weather data in feature set is kind of a cheating.  On the other hand, posted data include oil prices covering test period, so I guess the sponsors don't mind data leakage. Which is cool, as long as they realize that models using these kind of 'future telling' features might be less accurate in real life setting.",
    "237790": "Global Price of Cocoa from this link https://fred.stlouisfed.org/series/PCOCOUSDM\n\nGlobal Price of Bananas from this link  https://fred.stlouisfed.org/series/PBANSOPUSDM\n\nPopulation of city from this link  http://data.un.org/Data.aspx?d=POP&amp;f=tableCode%3A240",
    "234740": "Population Proyections\nI found this information from this link:\n\nhttp://www.ecuadorencifras.gob.ec/vdatos/\n",
    "261139": "More information about Holidays in Ecuador, from a local site:\n\nhttp://www.feriadosecuador.net/feriado/2014\n\nhttp://www.feriadosecuador.net/feriado/2015\n\nhttp://www.feriadosecuador.net/feriado/2016\n\nhttp://www.feriadosecuador.net/feriado/2017\n\n",
    "255991": "google trends https://trends.google.com/trends/explore?date=today%205-y&amp;q=Corporaci%C3%B3n%20Favorita",
    "234104": "Census data for the cities (if available).",
    "266559": "City &lt;-&gt; State (Provincia) mapping\n\nhttps://es.wikipedia.org/wiki/Anexo:Ciudades_de_Ecuador",
    "266166": "https://en.wikipedia.org/wiki/2018_FIFA_World_Cup_qualification_(CONMEBOL)",
    "265965": "not sure if it counts as external data, but I better list it - manual mapping from stores to regions (Costa, Sierra, Oriente) (wasn't able to find it as data so have to do myself)",
    "253226": "Could you clarify the rules with regards to external sources please. According to the rules: \"The source of any external data must be posted to the official competition forum prior to the First Submission Deadline.\" Does that mean in this case the deadline for posting external sources is January 15, 2018 @ 11:59 PM UTC?",
    "252802": "This is potentially a really dumb question, but if I'm using an external library like scipy, do I need to post that here? Would that count as \"external data\" under the definition given in the rules?\n\nThis part of the rules (below) confused me a bit, hence the question.\n\n*\"Use of publicly available external data is permitted, provided it does not pertain to Corporación Favorita nor to any entity in the same line of business as Corporación Favorita. **For example, you may use pre-trained networks or natural language tools/libraries**.\"*\n\n(Just to be on the safe side, pending an answer, I'm using scikit-learn, scipy, and numpy.)",
    "246545": "is there any way to get product descriptions or titles?",
    "240517": "There is a bunch of stuff in [worldbank][1] - mostly quarterly data, on country level. Note that is is posted usually with a few months delay, so it's best to shift the data 3 months.\n\n\n  [1]: https://data.worldbank.org/country/ecuador?view=chart",
    "238992": "\" you may not use data other than the Competition Data to develop and test your models and Submissions\"  -- Rule 7C.\n\nWhat is the purpose of having this External Data Thread, when the rule above exists?",
    "249821": "",
    "237239": ""
  }
}