{
  "id": 21223,
  "title": "Accuracy drop",
  "url": "/competitions/facebook-v-predicting-check-ins/discussion/21223",
  "author_name": "",
  "post_date": "2016-05-25T23:08:04.453Z",
  "votes": null,
  "comment_count": 1,
  "views": 546,
  "content": "<p>Hello everyone,</p>\n\n<p>This is my first competition here and right now I faced a surprise when doing my first submission. I turned the city into a grid and tried a RF classifier. I split the data for a square into 65% training data and 35% test data, randomly and tested the accuracy for squares in different positions in the city.</p>\n\n<p>It gave me results between 0.54 and 0.56 accuracy, depending on the place on the grid. Then I run it for the whole set (this only with one prediction for now). I was expecting a small drop when submiting the results, but it gave me 0.42. </p>\n\n<p>Is there any obvious reason for a drop like this, or it is just related with the variability of both sets?</p>\n\n<p>Thanks in advance</p>",
  "messages": [
    {
      "id": "121368",
      "postDate": "05/25/2016 23:08:04",
      "content": "<p>Hello everyone,</p>\n\n<p>This is my first competition here and right now I faced a surprise when doing my first submission. I turned the city into a grid and tried a RF classifier. I split the data for a square into 65% training data and 35% test data, randomly and tested the accuracy for squares in different positions in the city.</p>\n\n<p>It gave me results between 0.54 and 0.56 accuracy, depending on the place on the grid. Then I run it for the whole set (this only with one prediction for now). I was expecting a small drop when submiting the results, but it gave me 0.42. </p>\n\n<p>Is there any obvious reason for a drop like this, or it is just related with the variability of both sets?</p>\n\n<p>Thanks in advance</p>",
      "rawMarkdown": "Hello everyone,\r\n\r\nThis is my first competition here and right now I faced a surprise when doing my first submission. I turned the city into a grid and tried a RF classifier. I split the data for a square into 65% training data and 35% test data, randomly and tested the accuracy for squares in different positions in the city.\r\n\r\nIt gave me results between 0.54 and 0.56 accuracy, depending on the place on the grid. Then I run it for the whole set (this only with one prediction for now). I was expecting a small drop when submiting the results, but it gave me 0.42. \r\n\r\nIs there any obvious reason for a drop like this, or it is just related with the variability of both sets?\r\n\r\nThanks in advance",
      "votes": null
    },
    {
      "id": "121370",
      "postDate": "05/25/2016 23:44:06",
      "content": "<p>The training and test data are not split randomly.  Instead they are split by time with test data coming from dates after the training data.  So if you did your validation by randomly splitting the training data, that probably would have given you artificially high accuracy.  I would recommend splitting the training data using the time variable.</p>",
      "rawMarkdown": "The training and test data are not split randomly.  Instead they are split by time with test data coming from dates after the training data.  So if you did your validation by randomly splitting the training data, that probably would have given you artificially high accuracy.  I would recommend splitting the training data using the time variable.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 121370,
      "author_name": "breakfastpirate",
      "author_url": "",
      "post_date": "05/25/2016 23:44:06",
      "content": "<p>The training and test data are not split randomly.  Instead they are split by time with test data coming from dates after the training data.  So if you did your validation by randomly splitting the training data, that probably would have given you artificially high accuracy.  I would recommend splitting the training data using the time variable.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "121368": "Hello everyone,\r\n\r\nThis is my first competition here and right now I faced a surprise when doing my first submission. I turned the city into a grid and tried a RF classifier. I split the data for a square into 65% training data and 35% test data, randomly and tested the accuracy for squares in different positions in the city.\r\n\r\nIt gave me results between 0.54 and 0.56 accuracy, depending on the place on the grid. Then I run it for the whole set (this only with one prediction for now). I was expecting a small drop when submiting the results, but it gave me 0.42. \r\n\r\nIs there any obvious reason for a drop like this, or it is just related with the variability of both sets?\r\n\r\nThanks in advance",
    "121370": "The training and test data are not split randomly.  Instead they are split by time with test data coming from dates after the training data.  So if you did your validation by randomly splitting the training data, that probably would have given you artificially high accuracy.  I would recommend splitting the training data using the time variable."
  },
  "source": "meta"
}