{
  "id": 21843,
  "title": "One question about the overlapping grid cell in the public script",
  "url": "/competitions/facebook-v-predicting-check-ins/discussion/21843",
  "author_name": "",
  "post_date": "2016-06-22T17:06:14.253Z",
  "votes": null,
  "comment_count": 3,
  "views": 369,
  "content": "<p>I doubt there's some problem when we use overlapping grid cell to do predictions. The code in the PB script is like </p>\n\n<pre><code>def process_grid(df_train, df_test):\n&quot;&quot;&quot;\nIterates over all grid cells, aggregates the results\n&quot;&quot;&quot;\nsize = 10.0\nx_step = 0.5\ny_step = 0.25\n\nx_border_augment = 0.025    \ny_border_augment = 0.0125\n\npreds = np.zeros((df_test.shape[0], 3), dtype=int)\n\nfor i in range((int)(size/x_step)):\n\n    x_min = x_step * i\n    x_max = x_step * (i+1)\n    x_min = round(x_min, 4)\n    x_max = round(x_max, 4) \n    if x_max == size:\n        x_max = x_max + 0.001\n\n    df_col_train = df_train[(df_train['x'] &gt;= x_min-x_border_augment) &amp; (df_train['x'] &lt;       x_max+x_border_augment)]\n    df_col_test = df_test[(df_test['x'] &gt;= x_min) &amp; (df_test['x'] &lt; x_max)]\n\n    for j in range((int)(size/y_step)):\n        y_min = y_step * j\n        y_max = y_step * (j+1)\n        y_min = round(y_min, 4)\n        y_max = round(y_max, 4)   \n        if y_max == size:\n            y_max = y_max + 0.001\n\n        df_cell_train = df_col_train[(df_col_train['y'] &gt;= y_min-y_border_augment) &amp; (df_col_train['y'] &lt; y_max+y_border_augment)]\n        df_cell_test = df_col_test[(df_col_test['y'] &gt;= y_min) &amp; (df_col_test['y'] &lt; y_max)]\n\n        #Applying classifier to one grid cell\n        pred_labels, row_ids = process_one_cell(df_cell_train, df_cell_test)\n\n        #Updating predictions\n        preds[row_ids] = pred_labels\n</code></pre>\n\n<p>I understand the purpose of doing border augmentation. But when we do prediction for each cell, the test id in the overlapping part will be predicted for several times.  But the code will continuously updating the predictions. That means the predictions for the  test ids in the overlapping part are just based on the latest trained cell. \nI'm not sure if I understand it correctly. If I'm wrong, please point it out. Thanks </p>",
  "messages": [
    {
      "id": "124832",
      "postDate": "06/22/2016 17:06:14",
      "content": "<p>I doubt there's some problem when we use overlapping grid cell to do predictions. The code in the PB script is like </p>\n\n<pre><code>def process_grid(df_train, df_test):\n&quot;&quot;&quot;\nIterates over all grid cells, aggregates the results\n&quot;&quot;&quot;\nsize = 10.0\nx_step = 0.5\ny_step = 0.25\n\nx_border_augment = 0.025    \ny_border_augment = 0.0125\n\npreds = np.zeros((df_test.shape[0], 3), dtype=int)\n\nfor i in range((int)(size/x_step)):\n\n    x_min = x_step * i\n    x_max = x_step * (i+1)\n    x_min = round(x_min, 4)\n    x_max = round(x_max, 4) \n    if x_max == size:\n        x_max = x_max + 0.001\n\n    df_col_train = df_train[(df_train['x'] &gt;= x_min-x_border_augment) &amp; (df_train['x'] &lt;       x_max+x_border_augment)]\n    df_col_test = df_test[(df_test['x'] &gt;= x_min) &amp; (df_test['x'] &lt; x_max)]\n\n    for j in range((int)(size/y_step)):\n        y_min = y_step * j\n        y_max = y_step * (j+1)\n        y_min = round(y_min, 4)\n        y_max = round(y_max, 4)   \n        if y_max == size:\n            y_max = y_max + 0.001\n\n        df_cell_train = df_col_train[(df_col_train['y'] &gt;= y_min-y_border_augment) &amp; (df_col_train['y'] &lt; y_max+y_border_augment)]\n        df_cell_test = df_col_test[(df_col_test['y'] &gt;= y_min) &amp; (df_col_test['y'] &lt; y_max)]\n\n        #Applying classifier to one grid cell\n        pred_labels, row_ids = process_one_cell(df_cell_train, df_cell_test)\n\n        #Updating predictions\n        preds[row_ids] = pred_labels\n</code></pre>\n\n<p>I understand the purpose of doing border augmentation. But when we do prediction for each cell, the test id in the overlapping part will be predicted for several times.  But the code will continuously updating the predictions. That means the predictions for the  test ids in the overlapping part are just based on the latest trained cell. \nI'm not sure if I understand it correctly. If I'm wrong, please point it out. Thanks </p>",
      "rawMarkdown": "I doubt there's some problem when we use overlapping grid cell to do predictions. The code in the PB script is like \r\n\r\n    def process_grid(df_train, df_test):\r\n    \"\"\"\r\n    Iterates over all grid cells, aggregates the results\r\n    \"\"\"\r\n    size = 10.0\r\n    x_step = 0.5\r\n    y_step = 0.25\r\n    \r\n    x_border_augment = 0.025    \r\n    y_border_augment = 0.0125\r\n    \r\n    preds = np.zeros((df_test.shape[0], 3), dtype=int)\r\n\r\n    for i in range((int)(size/x_step)):\r\n        \r\n        x_min = x_step * i\r\n        x_max = x_step * (i+1)\r\n        x_min = round(x_min, 4)\r\n        x_max = round(x_max, 4) \r\n        if x_max == size:\r\n            x_max = x_max + 0.001\r\n            \r\n        df_col_train = df_train[(df_train['x'] >= x_min-x_border_augment) & (df_train['x'] <       x_max+x_border_augment)]\r\n        df_col_test = df_test[(df_test['x'] >= x_min) & (df_test['x'] < x_max)]\r\n\r\n        for j in range((int)(size/y_step)):\r\n            y_min = y_step * j\r\n            y_max = y_step * (j+1)\r\n            y_min = round(y_min, 4)\r\n            y_max = round(y_max, 4)   \r\n            if y_max == size:\r\n                y_max = y_max + 0.001\r\n                \r\n            df_cell_train = df_col_train[(df_col_train['y'] >= y_min-y_border_augment) & (df_col_train['y'] < y_max+y_border_augment)]\r\n            df_cell_test = df_col_test[(df_col_test['y'] >= y_min) & (df_col_test['y'] < y_max)]\r\n            \r\n            #Applying classifier to one grid cell\r\n            pred_labels, row_ids = process_one_cell(df_cell_train, df_cell_test)\r\n\r\n            #Updating predictions\r\n            preds[row_ids] = pred_labels\r\n\r\nI understand the purpose of doing border augmentation. But when we do prediction for each cell, the test id in the overlapping part will be predicted for several times.  But the code will continuously updating the predictions. That means the predictions for the  test ids in the overlapping part are just based on the latest trained cell. \r\nI'm not sure if I understand it correctly. If I'm wrong, please point it out. Thanks",
      "votes": null
    },
    {
      "id": "124836",
      "postDate": "06/22/2016 17:31:36",
      "content": "<p>As per my understanding, based on the physical location &amp; time-stamp, any row in the test dataset can only belong to 1 cell of the grid. Hence you predict for it only once. But to make it's prediction, you'll consider all points in the training set from that particular grid, as well as the points that lie in the augmented zones of the adjacent cells.  </p>",
      "rawMarkdown": "As per my understanding, based on the physical location & time-stamp, any row in the test dataset can only belong to 1 cell of the grid. Hence you predict for it only once. But to make it's prediction, you'll consider all points in the training set from that particular grid, as well as the points that lie in the augmented zones of the adjacent cells.",
      "votes": null
    },
    {
      "id": "124839",
      "postDate": "06/22/2016 17:48:01",
      "content": "<p>@FengLi</p>\n\n<p>Notice</p>\n\n<p><code>df_col_test = df_test[(df_test['x'] &gt;= x_min) &amp; (df_test['x'] &lt; x_max)]</code></p>\n\n<p><code>df_cell_test = df_col_test[(df_col_test['y'] &gt;= y_min) &amp; (df_col_test['y'] &lt; y_max)]</code></p>\n\n<p>The selection of test data doesn't include the augmented part.</p>",
      "rawMarkdown": "FengLi\r\n\r\nNotice\r\n\r\n`df_col_test = df_test[(df_test['x'] >= x_min) & (df_test['x'] < x_max)]`\r\n\r\n`df_cell_test = df_col_test[(df_col_test['y'] >= y_min) & (df_col_test['y'] < y_max)]`\r\n\r\nThe selection of test data doesn't include the augmented part.",
      "votes": null
    },
    {
      "id": "124867",
      "postDate": "06/22/2016 21:42:33",
      "content": "<p>@Siddharth Chandrakant <br>\n@Leon\nThank you.I just read the train part... It's my fault. </p>",
      "rawMarkdown": "Siddharth Chandrakant   \r\n@Leon\r\nThank you.I just read the train part... It's my fault.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 124836,
      "author_name": "sadz2201",
      "author_url": "",
      "post_date": "06/22/2016 17:31:36",
      "content": "<p>As per my understanding, based on the physical location &amp; time-stamp, any row in the test dataset can only belong to 1 cell of the grid. Hence you predict for it only once. But to make it's prediction, you'll consider all points in the training set from that particular grid, as well as the points that lie in the augmented zones of the adjacent cells.  </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124839,
      "author_name": "leonlu",
      "author_url": "",
      "post_date": "06/22/2016 17:48:01",
      "content": "<p>@FengLi</p>\n\n<p>Notice</p>\n\n<p><code>df_col_test = df_test[(df_test['x'] &gt;= x_min) &amp; (df_test['x'] &lt; x_max)]</code></p>\n\n<p><code>df_cell_test = df_col_test[(df_col_test['y'] &gt;= y_min) &amp; (df_col_test['y'] &lt; y_max)]</code></p>\n\n<p>The selection of test data doesn't include the augmented part.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124867,
      "author_name": "beedata",
      "author_url": "",
      "post_date": "06/22/2016 21:42:33",
      "content": "<p>@Siddharth Chandrakant <br>\n@Leon\nThank you.I just read the train part... It's my fault. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "124832": "I doubt there's some problem when we use overlapping grid cell to do predictions. The code in the PB script is like \r\n\r\n    def process_grid(df_train, df_test):\r\n    \"\"\"\r\n    Iterates over all grid cells, aggregates the results\r\n    \"\"\"\r\n    size = 10.0\r\n    x_step = 0.5\r\n    y_step = 0.25\r\n    \r\n    x_border_augment = 0.025    \r\n    y_border_augment = 0.0125\r\n    \r\n    preds = np.zeros((df_test.shape[0], 3), dtype=int)\r\n\r\n    for i in range((int)(size/x_step)):\r\n        \r\n        x_min = x_step * i\r\n        x_max = x_step * (i+1)\r\n        x_min = round(x_min, 4)\r\n        x_max = round(x_max, 4) \r\n        if x_max == size:\r\n            x_max = x_max + 0.001\r\n            \r\n        df_col_train = df_train[(df_train['x'] >= x_min-x_border_augment) & (df_train['x'] <       x_max+x_border_augment)]\r\n        df_col_test = df_test[(df_test['x'] >= x_min) & (df_test['x'] < x_max)]\r\n\r\n        for j in range((int)(size/y_step)):\r\n            y_min = y_step * j\r\n            y_max = y_step * (j+1)\r\n            y_min = round(y_min, 4)\r\n            y_max = round(y_max, 4)   \r\n            if y_max == size:\r\n                y_max = y_max + 0.001\r\n                \r\n            df_cell_train = df_col_train[(df_col_train['y'] >= y_min-y_border_augment) & (df_col_train['y'] < y_max+y_border_augment)]\r\n            df_cell_test = df_col_test[(df_col_test['y'] >= y_min) & (df_col_test['y'] < y_max)]\r\n            \r\n            #Applying classifier to one grid cell\r\n            pred_labels, row_ids = process_one_cell(df_cell_train, df_cell_test)\r\n\r\n            #Updating predictions\r\n            preds[row_ids] = pred_labels\r\n\r\nI understand the purpose of doing border augmentation. But when we do prediction for each cell, the test id in the overlapping part will be predicted for several times.  But the code will continuously updating the predictions. That means the predictions for the  test ids in the overlapping part are just based on the latest trained cell. \r\nI'm not sure if I understand it correctly. If I'm wrong, please point it out. Thanks",
    "124836": "As per my understanding, based on the physical location & time-stamp, any row in the test dataset can only belong to 1 cell of the grid. Hence you predict for it only once. But to make it's prediction, you'll consider all points in the training set from that particular grid, as well as the points that lie in the augmented zones of the adjacent cells.",
    "124839": "FengLi\r\n\r\nNotice\r\n\r\n`df_col_test = df_test[(df_test['x'] >= x_min) & (df_test['x'] < x_max)]`\r\n\r\n`df_cell_test = df_col_test[(df_col_test['y'] >= y_min) & (df_col_test['y'] < y_max)]`\r\n\r\nThe selection of test data doesn't include the augmented part.",
    "124867": "Siddharth Chandrakant   \r\n@Leon\r\nThank you.I just read the train part... It's my fault."
  },
  "source": "meta"
}