{
  "id": 11226,
  "title": "Number of observations in submission file Issue",
  "url": "/competitions/avazu-ctr-prediction/discussion/11226",
  "author_name": "",
  "post_date": "2014-12-13T18:54:36.940Z",
  "votes": null,
  "comment_count": 3,
  "views": 1008,
  "content": "<p>Hi,I have used following code for submission. My model name is &quot;clf&quot;. I have used only 2 variables here for illustration.</p>\n<p><code>data_test = pd.read_csv('test_rev2.csv', dtype={'id': str})</code></p>\n<p><code></code><code>X_test = data_test.ix[:,('site_id', 'device_ip')]</code><code></code></p>\n<p><code>proba = clf.predict_proba(X_test)<br>proba = pd.DataFrame(proba[:,1])<br>proba.columns = [&quot;click&quot;]</code><code></code></p>\n<p><code>score_data = pd.concat((data_test[&quot;id&quot;], proba), ignore_index=True , axis = 1)<br>score_data.columns = [&quot;id&quot;,&quot;click&quot;]</code><code></code></p>\n<p><code>score_data.iloc[0:4577464,:].to_csv(&quot;Submission.csv&quot;, index = False, dtype={'id': str})</code><code></code></p>\n<p>There are 4,769,401 obs in test data set. Why submission file require only 4,577,464 obs. Is there any requirement about which &quot;id&quot; should be in csv file?</p>\n<p>I am getting the following error:</p>\n<p>ERROR: Unable to find 4563178 required key values in the 'id' column ERROR: Unable to find the required key value '10000174058809263569' in the 'id' column ERROR: Unable to find the required key value '10000182526920855428' in the 'id' column ERROR: Unable to find the required key value '10000554139829213984' in the 'id' column ERROR: Unable to find the required key value '10001094637809798845' in the 'id' column ERROR: Unable to find the required key value '10001377041558670745' in the 'id' column ERROR: Unable to find the required key value '10001521204153353724' in the 'id' column ERROR: Unable to find the required key value '10001911056707023378' in the 'id' column ERROR: Unable to find the required key value '10001982898844213216' in the 'id' column ERROR: Unable to find the required key value '10002000217531288531' in the 'id' column.</p>",
  "messages": [
    {
      "id": "60156",
      "postDate": "12/13/2014 18:54:36",
      "content": "<p>Hi,I have used following code for submission. My model name is &quot;clf&quot;. I have used only 2 variables here for illustration.</p>\n<p><code>data_test = pd.read_csv('test_rev2.csv', dtype={'id': str})</code></p>\n<p><code></code><code>X_test = data_test.ix[:,('site_id', 'device_ip')]</code><code></code></p>\n<p><code>proba = clf.predict_proba(X_test)<br>proba = pd.DataFrame(proba[:,1])<br>proba.columns = [&quot;click&quot;]</code><code></code></p>\n<p><code>score_data = pd.concat((data_test[&quot;id&quot;], proba), ignore_index=True , axis = 1)<br>score_data.columns = [&quot;id&quot;,&quot;click&quot;]</code><code></code></p>\n<p><code>score_data.iloc[0:4577464,:].to_csv(&quot;Submission.csv&quot;, index = False, dtype={'id': str})</code><code></code></p>\n<p>There are 4,769,401 obs in test data set. Why submission file require only 4,577,464 obs. Is there any requirement about which &quot;id&quot; should be in csv file?</p>\n<p>I am getting the following error:</p>\n<p>ERROR: Unable to find 4563178 required key values in the 'id' column ERROR: Unable to find the required key value '10000174058809263569' in the 'id' column ERROR: Unable to find the required key value '10000182526920855428' in the 'id' column ERROR: Unable to find the required key value '10000554139829213984' in the 'id' column ERROR: Unable to find the required key value '10001094637809798845' in the 'id' column ERROR: Unable to find the required key value '10001377041558670745' in the 'id' column ERROR: Unable to find the required key value '10001521204153353724' in the 'id' column ERROR: Unable to find the required key value '10001911056707023378' in the 'id' column ERROR: Unable to find the required key value '10001982898844213216' in the 'id' column ERROR: Unable to find the required key value '10002000217531288531' in the 'id' column.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60160",
      "postDate": "12/13/2014 20:24:42",
      "content": "<p>Hi Vinod -</p>\n<p>My test file has&nbsp;4577465 ( Observations :&nbsp;4577464 and Header : 1) and submission file has the same Ids as in test. &nbsp;</p>\n<p><em>MacBook-Air-2:data dr$ wc -l test</em><br><em> 4577465 test</em><br><em>MacBook-Air-2:data dr$ wc -l submission.csv </em><br><em> 4577465 submission.csv</em></p>\n<p>Let me know if this helps.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60173",
      "postDate": "12/14/2014 04:12:50",
      "content": "<p>I am getting a similar error. Any luck with your issue?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60478",
      "postDate": "12/17/2014 13:36:38",
      "content": "<p>Download the data again. I did this. and it is working :)</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 60160,
      "author_name": "",
      "author_url": "",
      "post_date": "12/13/2014 20:24:42",
      "content": "<p>Hi Vinod -</p>\n<p>My test file has&nbsp;4577465 ( Observations :&nbsp;4577464 and Header : 1) and submission file has the same Ids as in test. &nbsp;</p>\n<p><em>MacBook-Air-2:data dr$ wc -l test</em><br><em> 4577465 test</em><br><em>MacBook-Air-2:data dr$ wc -l submission.csv </em><br><em> 4577465 submission.csv</em></p>\n<p>Let me know if this helps.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60173,
      "author_name": "pawan136929",
      "author_url": "",
      "post_date": "12/14/2014 04:12:50",
      "content": "<p>I am getting a similar error. Any luck with your issue?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60478,
      "author_name": "vinodg",
      "author_url": "",
      "post_date": "12/17/2014 13:36:38",
      "content": "<p>Download the data again. I did this. and it is working :)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "60156": "",
    "60160": "",
    "60173": "",
    "60478": ""
  },
  "source": "meta"
}