{
  "id": 9613,
  "title": "Is this the right way to calculate the score?",
  "url": "/competitions/avito-prohibited-content/discussion/9613",
  "author_name": "",
  "post_date": "2014-06-30T07:58:02.713Z",
  "votes": null,
  "comment_count": 1,
  "views": 1015,
  "content": "<p><code>from sklearn.metrics import average_precision_score</code></p>\n<p><code></code><code><br></code><code>def score(true, pred):<br>&nbsp; &nbsp;no_features = 0.0481 * len(true)<br>&nbsp; &nbsp;items = pred.argsort()[::-1][:no_features]</code></p>\n\n<p><code>&nbsp; &nbsp;return average_precision_score(true[items], pred[items])</code></p>",
  "messages": [
    {
      "id": "49829",
      "postDate": "06/30/2014 07:58:02",
      "content": "<p><code>from sklearn.metrics import average_precision_score</code></p>\n<p><code></code><code><br></code><code>def score(true, pred):<br>&nbsp; &nbsp;no_features = 0.0481 * len(true)<br>&nbsp; &nbsp;items = pred.argsort()[::-1][:no_features]</code></p>\n\n<p><code>&nbsp; &nbsp;return average_precision_score(true[items], pred[items])</code></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49865",
      "postDate": "06/30/2014 15:12:37",
      "content": "<p>I don't think so. Have you seen this thread: http://www.kaggle.com/c/avito-prohibited-content/forums/t/9600/cross-validation-ap-32500/49774#post49774 ?</p>\n<p>You should first randomly select 50% of True Values and then take the top 4.81% and calculate AP.</p>\n<p>I will post my code when I get home later.</p>\n<p><strong>Update:</strong></p>\n<p><code>def getAP(actual, predicted):<br> #constants:<br> SEED=123<br> AP_ON_LB_CALCULATED_AT = 32500.00<br> NUM_ROWS_IN_TEST_SET=1351243<br> PERC_OF_TEST_ROWS_USED_IN_LB=0.5<br> NUM_ROWS_IN_TEST_SET_USED_IN_LB=NUM_ROWS_IN_TEST_SET*PERC_OF_TEST_ROWS_USED_IN_LB</code></p>\n<p><code>PERCENT_OF_LB_TEST_ROWS_USED_FOR_AP =AP_ON_LB_CALCULATED_AT/NUM_ROWS_IN_TEST_SET_USED_IN_LB</code></p>\n<p><code>NUM_ROWS_TO_SAMPLE_FOR_AP_CALC = round(len(predicted)*PERC_OF_TEST_ROWS_USED_IN_LB)<br> NUM_ROWS_TO_SELECT_FOR_AP_CALC = round(NUM_ROWS_TO_SAMPLE_FOR_AP_CALC*PERCENT_OF_LB_TEST_ROWS_USED_FOR_AP)</code></p>\n<p><code>#select NUM_ROWS_TO_SAMPLE_FOR_AP_CALC % of predicted rows:<br> np.random.seed(SEED)<br> indices = np.random.permutation(len(predicted))[:NUM_ROWS_TO_SAMPLE_FOR_AP_CALC]</code></p>\n\n<p><code>actualSampled =actual[indices]<br> predictedSampled =predicted[indices]</code></p>\n<p><code><br> #select top NUM_ROWS_TO_SELECT_FOR_AP_CALC rows ordered by p(is_blocked=1)<br> predictedIndicesForAPCalc=predictedSampled.argsort()[::-1][:NUM_ROWS_TO_SELECT_FOR_AP_CALC]</code></p>\n<p><code>predictedForAPCalc = predictedSampled[predictedIndicesForAPCalc]<br> actualForAPCalc = actualSampled[predictedIndicesForAPCalc]</code></p>\n<p><code>ap =average_precision_score(actualForAPCalc,predictedForAPCalc)<br> return ap<br></code></p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 49865,
      "author_name": "sashikanthdareddy",
      "author_url": "",
      "post_date": "06/30/2014 15:12:37",
      "content": "<p>I don't think so. Have you seen this thread: http://www.kaggle.com/c/avito-prohibited-content/forums/t/9600/cross-validation-ap-32500/49774#post49774 ?</p>\n<p>You should first randomly select 50% of True Values and then take the top 4.81% and calculate AP.</p>\n<p>I will post my code when I get home later.</p>\n<p><strong>Update:</strong></p>\n<p><code>def getAP(actual, predicted):<br> #constants:<br> SEED=123<br> AP_ON_LB_CALCULATED_AT = 32500.00<br> NUM_ROWS_IN_TEST_SET=1351243<br> PERC_OF_TEST_ROWS_USED_IN_LB=0.5<br> NUM_ROWS_IN_TEST_SET_USED_IN_LB=NUM_ROWS_IN_TEST_SET*PERC_OF_TEST_ROWS_USED_IN_LB</code></p>\n<p><code>PERCENT_OF_LB_TEST_ROWS_USED_FOR_AP =AP_ON_LB_CALCULATED_AT/NUM_ROWS_IN_TEST_SET_USED_IN_LB</code></p>\n<p><code>NUM_ROWS_TO_SAMPLE_FOR_AP_CALC = round(len(predicted)*PERC_OF_TEST_ROWS_USED_IN_LB)<br> NUM_ROWS_TO_SELECT_FOR_AP_CALC = round(NUM_ROWS_TO_SAMPLE_FOR_AP_CALC*PERCENT_OF_LB_TEST_ROWS_USED_FOR_AP)</code></p>\n<p><code>#select NUM_ROWS_TO_SAMPLE_FOR_AP_CALC % of predicted rows:<br> np.random.seed(SEED)<br> indices = np.random.permutation(len(predicted))[:NUM_ROWS_TO_SAMPLE_FOR_AP_CALC]</code></p>\n\n<p><code>actualSampled =actual[indices]<br> predictedSampled =predicted[indices]</code></p>\n<p><code><br> #select top NUM_ROWS_TO_SELECT_FOR_AP_CALC rows ordered by p(is_blocked=1)<br> predictedIndicesForAPCalc=predictedSampled.argsort()[::-1][:NUM_ROWS_TO_SELECT_FOR_AP_CALC]</code></p>\n<p><code>predictedForAPCalc = predictedSampled[predictedIndicesForAPCalc]<br> actualForAPCalc = actualSampled[predictedIndicesForAPCalc]</code></p>\n<p><code>ap =average_precision_score(actualForAPCalc,predictedForAPCalc)<br> return ap<br></code></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "49829": "",
    "49865": ""
  },
  "source": "meta"
}