{
  "id": 424996,
  "title": "Maybe a Small Bug in Metric Derivation?",
  "url": "/competitions/cafa-5-protein-function-prediction/discussion/424996",
  "author_name": "AbaoJiang",
  "post_date": "2023-07-16T16:29:40.515000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hi everyone,</p>\n<p>When studying how the final score is derived, I observe there might exist a little bug in the <strong>normalization</strong> part. The code snippet is as follows (from line 150 to 152 in file <code>evaluation.py</code>),</p>\n<pre><code>\nmetrics[column] = np.divide(metrics[column], metrics[],\n                            out=np.zeros_like(metrics[column], dtype=), where=metrics[] &gt; )\n</code></pre>\n<p>The division is done for rows with <code>cov</code> larger than one, but the denominator is <code>wcov</code>. These two coverages are calculated with different supported terms. The formulas are shown as follows,</p>\n<h4>1. Coverage <code>cov</code> (from line 40 to 43 in file <code>evaluation.py</code>)</h4>\n<pre><code>p = solidify_prediction(pred.matrix[:, toi], tau)\nmetrics[i, ] = (p.(axis=) &gt; ).()\n</code></pre>\n<h4>2. Weighted coverage <code>wcov</code> (from line 66 to 69 in file <code>evaluation.py</code>)</h4>\n<pre><code>p = solidify_prediction(pred.matrix[:, toi], tau)  \nmetrics[i, ] = (p.(axis=) &gt; ).()\n</code></pre>\n<p>Though the code snippets seem the same, but the second one is actually derived with <code>toi_ia</code> (<em>i.e.,</em> terms with positive IA). If all predictions are terms with <strong>zero IA</strong>, the division can be successfully passed for entries with <strong>zero <code>wcov</code> and non-zero <code>cov</code></strong>, then the result (<em>i.e.,</em> WPR) is infinite.</p>\n<p>Maybe the code snippet of normalization should be adjusted to</p>\n<pre><code>\nmetrics[column] = np.divide(metrics[column], metrics[],\n                            out=np.zeros_like(metrics[column], dtype=), where=metrics[] &gt; )\n</code></pre>\n<p>If there's any misunderstanding, please let me know. Thanks a lot!<br>\n<a href=\"https://www.kaggle.com/iddux1\" target=\"_blank\">@iddux1</a> <a href=\"https://www.kaggle.com/claradepaolis\" target=\"_blank\">@claradepaolis</a> </p>",
  "messages": [
    {
      "id": 2346938,
      "postDate": "2023-07-16T16:29:40.517Z",
      "content": "<p>Hi everyone,</p>\n<p>When studying how the final score is derived, I observe there might exist a little bug in the <strong>normalization</strong> part. The code snippet is as follows (from line 150 to 152 in file <code>evaluation.py</code>),</p>\n<pre><code>\nmetrics[column] = np.divide(metrics[column], metrics[],\n                            out=np.zeros_like(metrics[column], dtype=), where=metrics[] &gt; )\n</code></pre>\n<p>The division is done for rows with <code>cov</code> larger than one, but the denominator is <code>wcov</code>. These two coverages are calculated with different supported terms. The formulas are shown as follows,</p>\n<h4>1. Coverage <code>cov</code> (from line 40 to 43 in file <code>evaluation.py</code>)</h4>\n<pre><code>p = solidify_prediction(pred.matrix[:, toi], tau)\nmetrics[i, ] = (p.(axis=) &gt; ).()\n</code></pre>\n<h4>2. Weighted coverage <code>wcov</code> (from line 66 to 69 in file <code>evaluation.py</code>)</h4>\n<pre><code>p = solidify_prediction(pred.matrix[:, toi], tau)  \nmetrics[i, ] = (p.(axis=) &gt; ).()\n</code></pre>\n<p>Though the code snippets seem the same, but the second one is actually derived with <code>toi_ia</code> (<em>i.e.,</em> terms with positive IA). If all predictions are terms with <strong>zero IA</strong>, the division can be successfully passed for entries with <strong>zero <code>wcov</code> and non-zero <code>cov</code></strong>, then the result (<em>i.e.,</em> WPR) is infinite.</p>\n<p>Maybe the code snippet of normalization should be adjusted to</p>\n<pre><code>\nmetrics[column] = np.divide(metrics[column], metrics[],\n                            out=np.zeros_like(metrics[column], dtype=), where=metrics[] &gt; )\n</code></pre>\n<p>If there's any misunderstanding, please let me know. Thanks a lot!<br>\n<a href=\"https://www.kaggle.com/iddux1\" target=\"_blank\">@iddux1</a> <a href=\"https://www.kaggle.com/claradepaolis\" target=\"_blank\">@claradepaolis</a> </p>",
      "rawMarkdown": "Hi everyone,\n\nWhen studying how the final score is derived, I observe there might exist a little bug in the **normalization** part. The code snippet is as follows (from line 150 to 152 in file `evaluation.py`),\n\n```python\n# Normalize by weighted cov\nmetrics[column] = np.divide(metrics[column], metrics[\"wcov\"],\n                            out=np.zeros_like(metrics[column], dtype='float'), where=metrics[\"cov\"] > 0)\n```\n\nThe division is done for rows with `cov` larger than one, but the denominator is `wcov`. These two coverages are calculated with different supported terms. The formulas are shown as follows,\n#### 1. Coverage `cov` (from line 40 to 43 in file `evaluation.py`)\n```python\np = solidify_prediction(pred.matrix[:, toi], tau)\nmetrics[i, 0] = (p.sum(axis=1) > 0).sum()\n```\n\n#### 2. Weighted coverage `wcov` (from line 66 to 69 in file `evaluation.py`)\n```python\np = solidify_prediction(pred.matrix[:, toi], tau)  # toi is actually toi_ia\nmetrics[i, 0] = (p.sum(axis=1) > 0).sum()\n```\n\nThough the code snippets seem the same, but the second one is actually derived with `toi_ia` (*i.e.,* terms with positive IA). If all predictions are terms with **zero IA**, the division can be successfully passed for entries with **zero `wcov` and non-zero `cov`**, then the result (*i.e.,* WPR) is infinite.\n\nMaybe the code snippet of normalization should be adjusted to\n```python\n# Normalize by weighted cov\nmetrics[column] = np.divide(metrics[column], metrics[\"wcov\"],\n                            out=np.zeros_like(metrics[column], dtype='float'), where=metrics[\"wcov\"] > 0)\n```\n\nIf there's any misunderstanding, please let me know. Thanks a lot!\n@iddux1 @claradepaolis ",
      "votes": 3
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2346938": "Hi everyone,\n\nWhen studying how the final score is derived, I observe there might exist a little bug in the **normalization** part. The code snippet is as follows (from line 150 to 152 in file `evaluation.py`),\n\n```python\n# Normalize by weighted cov\nmetrics[column] = np.divide(metrics[column], metrics[\"wcov\"],\n                            out=np.zeros_like(metrics[column], dtype='float'), where=metrics[\"cov\"] > 0)\n```\n\nThe division is done for rows with `cov` larger than one, but the denominator is `wcov`. These two coverages are calculated with different supported terms. The formulas are shown as follows,\n#### 1. Coverage `cov` (from line 40 to 43 in file `evaluation.py`)\n```python\np = solidify_prediction(pred.matrix[:, toi], tau)\nmetrics[i, 0] = (p.sum(axis=1) > 0).sum()\n```\n\n#### 2. Weighted coverage `wcov` (from line 66 to 69 in file `evaluation.py`)\n```python\np = solidify_prediction(pred.matrix[:, toi], tau)  # toi is actually toi_ia\nmetrics[i, 0] = (p.sum(axis=1) > 0).sum()\n```\n\nThough the code snippets seem the same, but the second one is actually derived with `toi_ia` (*i.e.,* terms with positive IA). If all predictions are terms with **zero IA**, the division can be successfully passed for entries with **zero `wcov` and non-zero `cov`**, then the result (*i.e.,* WPR) is infinite.\n\nMaybe the code snippet of normalization should be adjusted to\n```python\n# Normalize by weighted cov\nmetrics[column] = np.divide(metrics[column], metrics[\"wcov\"],\n                            out=np.zeros_like(metrics[column], dtype='float'), where=metrics[\"wcov\"] > 0)\n```\n\nIf there's any misunderstanding, please let me know. Thanks a lot!\n@iddux1 @claradepaolis "
  }
}