While working with a local copy of the CAFA evaluator, i noticed that it is relatively expensive computationally. Its runtime grows both with the number of score thresholds and with the number of ontology terms, since large parts of the evaluation operate on full protein × term matrices. As a result, local validation can become noticeably slow. Below are some tricks that substantially reduced local validation time in my setup (require to clone a copy of the repo and add some changes to the code).
\nThe reference evaluator computes both unweighted and weighted variants if IA file is available. this triggers two compute_metrics(...) calls per namespace, i.e., a second pass over the same threshold grid. As LB uses only weighted scres, the first simpliest patch would be to compute the weighted variant only - this already gives substantial speedup.
Feed already filtered predictions to the evaluator (btw, as far as I can tell from the code, max_terms=500 can still result in 501 retained terms per protein, due to a <= max_terms check in the parser).
Looking at the prediction parser, some operations still scan full ontology-sized rows (the code repeatedly calls count_nonzero on dense rows while incrementally filling them, which makes the cost scale with the total number of ontology terms rather than with the number of predicted terms). So the idea is to change code a bit - maintain a counter that is updated only when a cell transitions from zero to non-zero. In my setup, this alone resulted in roughly a 1.5x speedup in the parser.
Propagation thouth looks like the main bottleneck, and for speedup, i tried this:
np.where(ont.dag[:, i]) inside the loop), the children lists can be built once and reused);where, and there is no need to include the term itself in the aggregation since only zero entries are being updated anyway);Taken together, these (point 4) changes improved propagation speed around 1.1× in my case. A more impactful idea was simple parallelization: splitting the matrix by rows and letting each process handle its own block while iterating over the ontology order locally. For BP ontology, this reduced propagation time by almost 2x.
\n5.On my machine, parsing the train_terms.tsv as ground truth took around 25 sec, and it happens on every evaluator call. But for local workflows, it is much more efficient to parse the ground truth once, serialize the processed representation, and pass it directly to the scorer instead
\nAlltogether the code changes above (1-5) resulted in an overall speedup of roughly 1.6x in my setup (~30 sec for all the steps the cafaeval does, except parsing GT, with th_step=0.01). this still felt slow for iterative local validation. Besides, the multiprocessing logic in the repo scoring function can be somewhat fragile. In certain combinations of th_step and n_cpu, worker processes may stall or hang locally (I was not able to fully diagnose or resolve this).
\nBUT! I found that a much much larger speedup came from changing the algorithmic shape of the computation itself. Instead of operating on full ontology-sized matrices padded with zeros, the idea is to work only with the predicted terms and corresponding ground-truth entries (a sparse-style pass). Replacing the dense logic with such an approach and implementing the core loops with numba resulted in an order-of-magnitude speedup (well over 10 times speeedup, WOW), so highly recommend!
\nI am curious how others handle local validation here. Do you run the full evaluator locally, or do you rely on some faster proxy metric that correlates well with the leaderboard score?
", "messages": [ { "id": "3381352", "postDate": "12/24/2025 09:46:00", "content": "While working with a local copy of the CAFA evaluator, i noticed that it is relatively expensive computationally. Its runtime grows both with the number of score thresholds and with the number of ontology terms, since large parts of the evaluation operate on full protein × term matrices. As a result, local validation can become noticeably slow. Below are some tricks that substantially reduced local validation time in my setup (require to clone a copy of the repo and add some changes to the code).
\nThe reference evaluator computes both unweighted and weighted variants if IA file is available. this triggers two compute_metrics(...) calls per namespace, i.e., a second pass over the same threshold grid. As LB uses only weighted scres, the first simpliest patch would be to compute the weighted variant only - this already gives substantial speedup.
Feed already filtered predictions to the evaluator (btw, as far as I can tell from the code, max_terms=500 can still result in 501 retained terms per protein, due to a <= max_terms check in the parser).
Looking at the prediction parser, some operations still scan full ontology-sized rows (the code repeatedly calls count_nonzero on dense rows while incrementally filling them, which makes the cost scale with the total number of ontology terms rather than with the number of predicted terms). So the idea is to change code a bit - maintain a counter that is updated only when a cell transitions from zero to non-zero. In my setup, this alone resulted in roughly a 1.5x speedup in the parser.
Propagation thouth looks like the main bottleneck, and for speedup, i tried this:
np.where(ont.dag[:, i]) inside the loop), the children lists can be built once and reused);where, and there is no need to include the term itself in the aggregation since only zero entries are being updated anyway);Taken together, these (point 4) changes improved propagation speed around 1.1× in my case. A more impactful idea was simple parallelization: splitting the matrix by rows and letting each process handle its own block while iterating over the ontology order locally. For BP ontology, this reduced propagation time by almost 2x.
\n5.On my machine, parsing the train_terms.tsv as ground truth took around 25 sec, and it happens on every evaluator call. But for local workflows, it is much more efficient to parse the ground truth once, serialize the processed representation, and pass it directly to the scorer instead
\nAlltogether the code changes above (1-5) resulted in an overall speedup of roughly 1.6x in my setup (~30 sec for all the steps the cafaeval does, except parsing GT, with th_step=0.01). this still felt slow for iterative local validation. Besides, the multiprocessing logic in the repo scoring function can be somewhat fragile. In certain combinations of th_step and n_cpu, worker processes may stall or hang locally (I was not able to fully diagnose or resolve this).
\nBUT! I found that a much much larger speedup came from changing the algorithmic shape of the computation itself. Instead of operating on full ontology-sized matrices padded with zeros, the idea is to work only with the predicted terms and corresponding ground-truth entries (a sparse-style pass). Replacing the dense logic with such an approach and implementing the core loops with numba resulted in an order-of-magnitude speedup (well over 10 times speeedup, WOW), so highly recommend!
\nI am curious how others handle local validation here. Do you run the full evaluator locally, or do you rely on some faster proxy metric that correlates well with the leaderboard score?
", "rawMarkdown": "While working with a local copy of the CAFA evaluator, i noticed that it is relatively expensive computationally. Its runtime grows both with the number of score thresholds and with the number of ontology terms, since large parts of the evaluation operate on full protein × term matrices. As a result, local validation can become noticeably slow. Below are some tricks that substantially reduced local validation time in my setup (require to clone a copy of the repo and add some changes to the code).\n\n1. The reference evaluator computes both unweighted and weighted variants if IA file is available. this triggers two `compute_metrics(...)` calls per namespace, i.e., a second pass over the same threshold grid. As LB uses only weighted scres, the first simpliest patch would be to compute the weighted variant only - this already gives substantial speedup.\n\n2. Feed already filtered predictions to the evaluator (btw, as far as I can tell from the code, max_terms=500 can still result in 501 retained terms per protein, due to a <= max_terms check in the parser).\n\n3. Looking at the prediction parser, some operations still scan full ontology-sized rows (the code repeatedly calls `count_nonzero` on dense rows while incrementally filling them, which makes the cost scale with the total number of ontology terms rather than with the number of predicted terms). So the idea is to change code a bit - maintain a counter that is updated only when a cell transitions from zero to non-zero. In my setup, this alone resulted in roughly a 1.5x speedup in the parser.\n\n4. Propagation thouth looks like the main bottleneck, and for speedup, i tried this:\n- cache children lists for each term (instead of repeatedly searching for children (via `np.where(ont.dag[:, i])` inside the loop), the children lists can be built once and reused);\n- for fill-mode propagation (used for LB for scoring), restrict row updates to rows where the current term is zero (a fast index-based lookup of zero rows is cheaper than a full `where`, and there is no need to include the term itself in the aggregation since only zero entries are being updated anyway);\n- replace checks based on summation with boolean existence checks for non-zero values, and avoid array element deletion in favor of slicing.\n\nTaken together, these (point 4) changes improved propagation speed around 1.1× in my case. A more impactful idea was simple parallelization: splitting the matrix by rows and letting each process handle its own block while iterating over the ontology order locally. For BP ontology, this reduced propagation time by almost 2x.\n\n5.On my machine, parsing the train_terms.tsv as ground truth took around 25 sec, and it happens on every evaluator call. But for local workflows, it is much more efficient to parse the ground truth once, serialize the processed representation, and pass it directly to the scorer instead\n\nAlltogether the code changes above (1-5) resulted in an overall speedup of roughly 1.6x in my setup **(~30 sec for all the steps the cafaeval does, except parsing GT, with th_step=0.01)**. this still felt slow for iterative local validation. Besides, the multiprocessing logic in the repo scoring function can be somewhat fragile. In certain combinations of th_step and n_cpu, worker processes may stall or hang locally (I was not able to fully diagnose or resolve this).\n\nBUT! I found that a much much larger speedup came from changing the algorithmic shape of the computation itself. Instead of operating on full ontology-sized matrices padded with zeros, the idea is to work only with the predicted terms and corresponding ground-truth entries (a sparse-style pass). Replacing the dense logic with such an approach and implementing the core loops with numba resulted in an order-of-magnitude speedup (**well over 10 times speeedup, WOW**), so highly recommend!\n\nI am curious how others handle local validation here. Do you run the full evaluator locally, or do you rely on some faster proxy metric that correlates well with the leaderboard score?", "votes": null }, { "id": "3381402", "postDate": "12/24/2025 13:36:11", "content": "I only score ~1000 terms per branch, so I precompute the constant FN from the remaining terms and compute F-max only over the scored set. I also skip propagation for local CV, this makes local validation much faster
", "rawMarkdown": "I only score ~1000 terms per branch, so I precompute the constant FN from the remaining terms and compute F-max only over the scored set. I also skip propagation for local CV, this makes local validation much faster", "votes": null }, { "id": "3381516", "postDate": "12/24/2025 19:49:25", "content": "Thanks! Does this correlate with LB? BTW, many teams on LB use codes like ‘INT3405E2’ in team names, just like yours, I got curious what they stand for ?
", "rawMarkdown": "Thanks! Does this correlate with LB? BTW, many teams on LB use codes like ‘INT3405E2’ in team names, just like yours, I got curious what they stand for ?", "votes": null }, { "id": "3381607", "postDate": "12/25/2025 02:46:55", "content": "INT3405E2 is my ML class code, and my professor used a Kaggle competition as the final exam. My CV seems to correlate fairly well with the LB, often being slightly higher
", "rawMarkdown": "INT3405E2 is my ML class code, and my professor used a Kaggle competition as the final exam. My CV seems to correlate fairly well with the LB, often being slightly higher", "votes": null }, { "id": "3382083", "postDate": "12/26/2025 12:32:39", "content": "Ah, I see, thanks for clarifying! Good luck with your exam!
", "rawMarkdown": "Ah, I see, thanks for clarifying! Good luck with your exam!", "votes": null }, { "id": "3383493", "postDate": "12/30/2025 09:29:52", "content": "Nice tips! Thank you very much
", "rawMarkdown": "Nice tips! Thank you very much", "votes": null }, { "id": "3394946", "postDate": "01/21/2026 22:54:45", "content": "How much time does your validation and submission pipeline takes.
", "rawMarkdown": "How much time does your validation and submission pipeline takes.", "votes": null }, { "id": "3395758", "postDate": "01/23/2026 16:13:28", "content": "in average the full scoring pipeline takes around 30 sec per all 3 aspects.
", "rawMarkdown": "in average the full scoring pipeline takes around 30 sec per all 3 aspects.", "votes": null }, { "id": "3440122", "postDate": "04/12/2026 01:00:11", "content": "I’ll need to run a lot of evaluations, so everything you mentioned is extremely valuable.
\nThanks a lot.
", "rawMarkdown": "I’ll need to run a lot of evaluations, so everything you mentioned is extremely valuable.\n\nThanks a lot.", "votes": null }, { "id": "3440732", "postDate": "04/12/2026 23:01:28", "content": "I've been running the evaluator on a tight inner loop for local validation over the last few weeks (iterating on an ensemble for CAFA 6) and hit exactly the pain points you describe — the full threshold sweep on dense protein × term matrices was making each experiment too slow to keep the cycle short. Your post, and especially your speedup-local branch (https://github.com/T0chka/CAFA-evaluator-PK-speedup), pointed me in the right direction before I had to figure it out myself, so thank you for that.
I ended up packaging my changes as a small fork, cafaeval-protea (https://github.com/frapercan/cafaeval-protea) (GPLv3), so I could pin it and re-run everything against a frozen upstream oracle before trusting any number. It starts from cherry-picking your five commits bit-exact (same authorship in git log), and then adds four sparse kernels on top: the NK/LK confusion matrix as a single np.bincount scatter + right-to-left cumsum, the PK variant with a boolean exclude mask so the per-protein toi lists are never built, a scatter-based push-up for propagation that skips every term with no predictions, and finally a PyArrow-backed rewrite of pred_parser because once the kernels were fast the Python line loop suddenly owned ~75% of wall time.
On a 4.45M-row real prediction file it runs NK in 0.45 s and PK in 0.65 s, versus 1.4 s / 3.0 s unmodified upstream. That's 3–5× end-to-end, which is real for my loop but still well short of the 10× you reported — and after profiling I can see exactly why: the kernels are <15 ms now, propagate is ~40 ms, and the parser is what's left to kill. The next honest step is probably to go fully sparse end-to-end — working only on the predicted terms and the corresponding ground-truth entries, the way you described — instead of more numpy tricks on dense matrices.
\nEither way, the direction you sketched in this thread has been load-bearing for my setup. Really appreciate you writing it up.
", "rawMarkdown": "I've been running the evaluator on a tight inner loop for local validation over the last few weeks (iterating on an ensemble for CAFA 6) and hit exactly the pain points you describe — the full threshold sweep on dense protein × term matrices was making each experiment too slow to keep the cycle short. Your post, and especially your `speedup-local` branch (https://github.com/T0chka/CAFA-evaluator-PK-speedup), pointed me in the right direction before I had to figure it out myself, so thank you for that.\n\nI ended up packaging my changes as a small fork, cafaeval-protea (https://github.com/frapercan/cafaeval-protea) (GPLv3), so I could pin it and re-run everything against a frozen upstream oracle before trusting any number. It starts from cherry-picking your five commits bit-exact (same authorship in `git log`), and then adds four sparse kernels on top: the NK/LK confusion matrix as a single `np.bincount` scatter + right-to-left cumsum, the PK variant with a boolean exclude mask so the per-protein `toi` lists are never built, a scatter-based push-up for propagation that skips every term with no predictions, and finally a PyArrow-backed rewrite of `pred_parser` because once the kernels were fast the Python line loop suddenly owned ~75% of wall time.\n\nOn a 4.45M-row real prediction file it runs NK in 0.45 s and PK in 0.65 s, versus 1.4 s / 3.0 s unmodified upstream. That's 3–5× end-to-end, which is real for my loop but still well short of the 10× you reported — and after profiling I can see exactly why: the kernels are <15 ms now, propagate is ~40 ms, and the parser is what's left to kill. The next honest step is probably to go fully sparse end-to-end — working only on the predicted terms and the corresponding ground-truth entries, the way you described — instead of more numpy tricks on dense matrices.\n\nEither way, the direction you sketched in this thread has been load-bearing for my setup. Really appreciate you writing it up.", "votes": null }, { "id": "3440897", "postDate": "04/13/2026 08:14:31", "content": "Really glad the post was useful! and, honestly, a bit embarrassed on my side 😅, because I never properly finished that branch.
\nWhat I actually use now pushes almost all evaluator-side preparation out of the inner loop. I prebuild the ontology and annotation side once as aspect-specific artifacts, then keep the local path in sparse/CSR form for candidate selection, NOT/known-term exclusion, and score propagation up the DAG with numba. I also made a small notebook showing that propagation kernel in isolation - basically a fast max-propagation over the GO DAG using CSR adjacency + Numba, because that step used to be one of the main practical bottlenecks for local validation.
\nMy local final scorer itself is also no longer the original dense-style threshold sweep over ontology-sized matrices. I rewrote the weighted F-score around aligned top-k predictions against CSR ground truth, so in practice the metric computation stopped being the bottleneck.
\nSo sorry that I left the public branch half-finished. It was a useful intermediate step, but once I had a local path that reproduced the reference cafaeval scores exactly, it turned out to be simpler to write a local evaluator than to keep optimizing the original dense code. Still, I’m very happy the branch was useful!
", "rawMarkdown": "Really glad the post was useful! and, honestly, a bit embarrassed on my side 😅, because I never properly finished that branch.\n\nWhat I actually use now pushes almost all evaluator-side preparation out of the inner loop. I prebuild the ontology and annotation side once as aspect-specific artifacts, then keep the local path in sparse/CSR form for candidate selection, NOT/known-term exclusion, and score propagation up the DAG with numba. I also made a small [notebook](https://www.kaggle.com/code/antoninadolgorukova/cafa6-super-fast-go-propagation-with-numba) showing that propagation kernel in isolation - basically a fast max-propagation over the GO DAG using CSR adjacency + Numba, because that step used to be one of the main practical bottlenecks for local validation.\n\nMy local final scorer itself is also no longer the original dense-style threshold sweep over ontology-sized matrices. I rewrote the weighted F-score around aligned top-k predictions against CSR ground truth, so in practice the metric computation stopped being the bottleneck.\n\nSo sorry that I left the public branch half-finished. It was a useful intermediate step, but once I had a local path that reproduced the reference cafaeval scores exactly, it turned out to be simpler to write a local evaluator than to keep optimizing the original dense code. Still, I’m very happy the branch was useful!", "votes": null }, { "id": "3441032", "postDate": "04/13/2026 12:06:44", "content": "No worries at all — the branch was already more than enough to point me in the right direction! DAG propagation was something I'd been meaning to understand properly for a while, but honestly it wasn't the bottleneck in my setup so I'd never really dug in. Now that I actually did, a lot of things clicked into place, and I picked up a fair amount along the way — so if anything, your post gave me a good excuse to finally sit with it. Good luck with the competition!
", "rawMarkdown": "No worries at all — the branch was already more than enough to point me in the right direction! DAG propagation was something I'd been meaning to understand properly for a while, but honestly it wasn't the bottleneck in my setup so I'd never really dug in. Now that I actually did, a lot of things clicked into place, and I picked up a fair amount along the way — so if anything, your post gave me a good excuse to finally sit with it. Good luck with the competition!", "votes": null } ], "comments": [ { "id": 3381402, "author_name": "nguyncdngs", "author_url": "", "post_date": "12/24/2025 13:36:11", "content": "I only score ~1000 terms per branch, so I precompute the constant FN from the remaining terms and compute F-max only over the scored set. I also skip propagation for local CV, this makes local validation much faster
", "votes": null, "replies": [ { "id": 3381516, "author_name": "antoninadolgorukova", "author_url": "", "post_date": "12/24/2025 19:49:25", "content": "Thanks! Does this correlate with LB? BTW, many teams on LB use codes like ‘INT3405E2’ in team names, just like yours, I got curious what they stand for ?
", "votes": null, "replies": [ { "id": 3381607, "author_name": "nguyncdngs", "author_url": "", "post_date": "12/25/2025 02:46:55", "content": "INT3405E2 is my ML class code, and my professor used a Kaggle competition as the final exam. My CV seems to correlate fairly well with the LB, often being slightly higher
", "votes": null, "replies": [ { "id": 3382083, "author_name": "antoninadolgorukova", "author_url": "", "post_date": "12/26/2025 12:32:39", "content": "Ah, I see, thanks for clarifying! Good luck with your exam!
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3383493, "author_name": "sixjune", "author_url": "", "post_date": "12/30/2025 09:29:52", "content": "Nice tips! Thank you very much
", "votes": null, "replies": [] }, { "id": 3394946, "author_name": "vyankteshdwivedi", "author_url": "", "post_date": "01/21/2026 22:54:45", "content": "How much time does your validation and submission pipeline takes.
", "votes": null, "replies": [ { "id": 3395758, "author_name": "antoninadolgorukova", "author_url": "", "post_date": "01/23/2026 16:13:28", "content": "in average the full scoring pipeline takes around 30 sec per all 3 aspects.
", "votes": null, "replies": [] } ] }, { "id": 3440122, "author_name": "xaxipirulazo", "author_url": "", "post_date": "04/12/2026 01:00:11", "content": "I’ll need to run a lot of evaluations, so everything you mentioned is extremely valuable.
\nThanks a lot.
", "votes": null, "replies": [] }, { "id": 3440732, "author_name": "xaxipirulazo", "author_url": "", "post_date": "04/12/2026 23:01:28", "content": "I've been running the evaluator on a tight inner loop for local validation over the last few weeks (iterating on an ensemble for CAFA 6) and hit exactly the pain points you describe — the full threshold sweep on dense protein × term matrices was making each experiment too slow to keep the cycle short. Your post, and especially your speedup-local branch (https://github.com/T0chka/CAFA-evaluator-PK-speedup), pointed me in the right direction before I had to figure it out myself, so thank you for that.
I ended up packaging my changes as a small fork, cafaeval-protea (https://github.com/frapercan/cafaeval-protea) (GPLv3), so I could pin it and re-run everything against a frozen upstream oracle before trusting any number. It starts from cherry-picking your five commits bit-exact (same authorship in git log), and then adds four sparse kernels on top: the NK/LK confusion matrix as a single np.bincount scatter + right-to-left cumsum, the PK variant with a boolean exclude mask so the per-protein toi lists are never built, a scatter-based push-up for propagation that skips every term with no predictions, and finally a PyArrow-backed rewrite of pred_parser because once the kernels were fast the Python line loop suddenly owned ~75% of wall time.
On a 4.45M-row real prediction file it runs NK in 0.45 s and PK in 0.65 s, versus 1.4 s / 3.0 s unmodified upstream. That's 3–5× end-to-end, which is real for my loop but still well short of the 10× you reported — and after profiling I can see exactly why: the kernels are <15 ms now, propagate is ~40 ms, and the parser is what's left to kill. The next honest step is probably to go fully sparse end-to-end — working only on the predicted terms and the corresponding ground-truth entries, the way you described — instead of more numpy tricks on dense matrices.
\nEither way, the direction you sketched in this thread has been load-bearing for my setup. Really appreciate you writing it up.
", "votes": null, "replies": [ { "id": 3440897, "author_name": "antoninadolgorukova", "author_url": "", "post_date": "04/13/2026 08:14:31", "content": "Really glad the post was useful! and, honestly, a bit embarrassed on my side 😅, because I never properly finished that branch.
\nWhat I actually use now pushes almost all evaluator-side preparation out of the inner loop. I prebuild the ontology and annotation side once as aspect-specific artifacts, then keep the local path in sparse/CSR form for candidate selection, NOT/known-term exclusion, and score propagation up the DAG with numba. I also made a small notebook showing that propagation kernel in isolation - basically a fast max-propagation over the GO DAG using CSR adjacency + Numba, because that step used to be one of the main practical bottlenecks for local validation.
\nMy local final scorer itself is also no longer the original dense-style threshold sweep over ontology-sized matrices. I rewrote the weighted F-score around aligned top-k predictions against CSR ground truth, so in practice the metric computation stopped being the bottleneck.
\nSo sorry that I left the public branch half-finished. It was a useful intermediate step, but once I had a local path that reproduced the reference cafaeval scores exactly, it turned out to be simpler to write a local evaluator than to keep optimizing the original dense code. Still, I’m very happy the branch was useful!
", "votes": null, "replies": [ { "id": 3441032, "author_name": "xaxipirulazo", "author_url": "", "post_date": "04/13/2026 12:06:44", "content": "No worries at all — the branch was already more than enough to point me in the right direction! DAG propagation was something I'd been meaning to understand properly for a while, but honestly it wasn't the bottleneck in my setup so I'd never really dug in. Now that I actually did, a lot of things clicked into place, and I picked up a fair amount along the way — so if anything, your post gave me a good excuse to finally sit with it. Good luck with the competition!
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3381352": "While working with a local copy of the CAFA evaluator, i noticed that it is relatively expensive computationally. Its runtime grows both with the number of score thresholds and with the number of ontology terms, since large parts of the evaluation operate on full protein × term matrices. As a result, local validation can become noticeably slow. Below are some tricks that substantially reduced local validation time in my setup (require to clone a copy of the repo and add some changes to the code).\n\n1. The reference evaluator computes both unweighted and weighted variants if IA file is available. this triggers two `compute_metrics(...)` calls per namespace, i.e., a second pass over the same threshold grid. As LB uses only weighted scres, the first simpliest patch would be to compute the weighted variant only - this already gives substantial speedup.\n\n2. Feed already filtered predictions to the evaluator (btw, as far as I can tell from the code, max_terms=500 can still result in 501 retained terms per protein, due to a <= max_terms check in the parser).\n\n3. Looking at the prediction parser, some operations still scan full ontology-sized rows (the code repeatedly calls `count_nonzero` on dense rows while incrementally filling them, which makes the cost scale with the total number of ontology terms rather than with the number of predicted terms). So the idea is to change code a bit - maintain a counter that is updated only when a cell transitions from zero to non-zero. In my setup, this alone resulted in roughly a 1.5x speedup in the parser.\n\n4. Propagation thouth looks like the main bottleneck, and for speedup, i tried this:\n- cache children lists for each term (instead of repeatedly searching for children (via `np.where(ont.dag[:, i])` inside the loop), the children lists can be built once and reused);\n- for fill-mode propagation (used for LB for scoring), restrict row updates to rows where the current term is zero (a fast index-based lookup of zero rows is cheaper than a full `where`, and there is no need to include the term itself in the aggregation since only zero entries are being updated anyway);\n- replace checks based on summation with boolean existence checks for non-zero values, and avoid array element deletion in favor of slicing.\n\nTaken together, these (point 4) changes improved propagation speed around 1.1× in my case. A more impactful idea was simple parallelization: splitting the matrix by rows and letting each process handle its own block while iterating over the ontology order locally. For BP ontology, this reduced propagation time by almost 2x.\n\n5.On my machine, parsing the train_terms.tsv as ground truth took around 25 sec, and it happens on every evaluator call. But for local workflows, it is much more efficient to parse the ground truth once, serialize the processed representation, and pass it directly to the scorer instead\n\nAlltogether the code changes above (1-5) resulted in an overall speedup of roughly 1.6x in my setup **(~30 sec for all the steps the cafaeval does, except parsing GT, with th_step=0.01)**. this still felt slow for iterative local validation. Besides, the multiprocessing logic in the repo scoring function can be somewhat fragile. In certain combinations of th_step and n_cpu, worker processes may stall or hang locally (I was not able to fully diagnose or resolve this).\n\nBUT! I found that a much much larger speedup came from changing the algorithmic shape of the computation itself. Instead of operating on full ontology-sized matrices padded with zeros, the idea is to work only with the predicted terms and corresponding ground-truth entries (a sparse-style pass). Replacing the dense logic with such an approach and implementing the core loops with numba resulted in an order-of-magnitude speedup (**well over 10 times speeedup, WOW**), so highly recommend!\n\nI am curious how others handle local validation here. Do you run the full evaluator locally, or do you rely on some faster proxy metric that correlates well with the leaderboard score?", "3381402": "I only score ~1000 terms per branch, so I precompute the constant FN from the remaining terms and compute F-max only over the scored set. I also skip propagation for local CV, this makes local validation much faster", "3381516": "Thanks! Does this correlate with LB? BTW, many teams on LB use codes like ‘INT3405E2’ in team names, just like yours, I got curious what they stand for ?", "3381607": "INT3405E2 is my ML class code, and my professor used a Kaggle competition as the final exam. My CV seems to correlate fairly well with the LB, often being slightly higher", "3382083": "Ah, I see, thanks for clarifying! Good luck with your exam!", "3383493": "Nice tips! Thank you very much", "3394946": "How much time does your validation and submission pipeline takes.", "3395758": "in average the full scoring pipeline takes around 30 sec per all 3 aspects.", "3440122": "I’ll need to run a lot of evaluations, so everything you mentioned is extremely valuable.\n\nThanks a lot.", "3440732": "I've been running the evaluator on a tight inner loop for local validation over the last few weeks (iterating on an ensemble for CAFA 6) and hit exactly the pain points you describe — the full threshold sweep on dense protein × term matrices was making each experiment too slow to keep the cycle short. Your post, and especially your `speedup-local` branch (https://github.com/T0chka/CAFA-evaluator-PK-speedup), pointed me in the right direction before I had to figure it out myself, so thank you for that.\n\nI ended up packaging my changes as a small fork, cafaeval-protea (https://github.com/frapercan/cafaeval-protea) (GPLv3), so I could pin it and re-run everything against a frozen upstream oracle before trusting any number. It starts from cherry-picking your five commits bit-exact (same authorship in `git log`), and then adds four sparse kernels on top: the NK/LK confusion matrix as a single `np.bincount` scatter + right-to-left cumsum, the PK variant with a boolean exclude mask so the per-protein `toi` lists are never built, a scatter-based push-up for propagation that skips every term with no predictions, and finally a PyArrow-backed rewrite of `pred_parser` because once the kernels were fast the Python line loop suddenly owned ~75% of wall time.\n\nOn a 4.45M-row real prediction file it runs NK in 0.45 s and PK in 0.65 s, versus 1.4 s / 3.0 s unmodified upstream. That's 3–5× end-to-end, which is real for my loop but still well short of the 10× you reported — and after profiling I can see exactly why: the kernels are <15 ms now, propagate is ~40 ms, and the parser is what's left to kill. The next honest step is probably to go fully sparse end-to-end — working only on the predicted terms and the corresponding ground-truth entries, the way you described — instead of more numpy tricks on dense matrices.\n\nEither way, the direction you sketched in this thread has been load-bearing for my setup. Really appreciate you writing it up.", "3440897": "Really glad the post was useful! and, honestly, a bit embarrassed on my side 😅, because I never properly finished that branch.\n\nWhat I actually use now pushes almost all evaluator-side preparation out of the inner loop. I prebuild the ontology and annotation side once as aspect-specific artifacts, then keep the local path in sparse/CSR form for candidate selection, NOT/known-term exclusion, and score propagation up the DAG with numba. I also made a small [notebook](https://www.kaggle.com/code/antoninadolgorukova/cafa6-super-fast-go-propagation-with-numba) showing that propagation kernel in isolation - basically a fast max-propagation over the GO DAG using CSR adjacency + Numba, because that step used to be one of the main practical bottlenecks for local validation.\n\nMy local final scorer itself is also no longer the original dense-style threshold sweep over ontology-sized matrices. I rewrote the weighted F-score around aligned top-k predictions against CSR ground truth, so in practice the metric computation stopped being the bottleneck.\n\nSo sorry that I left the public branch half-finished. It was a useful intermediate step, but once I had a local path that reproduced the reference cafaeval scores exactly, it turned out to be simpler to write a local evaluator than to keep optimizing the original dense code. Still, I’m very happy the branch was useful!", "3441032": "No worries at all — the branch was already more than enough to point me in the right direction! DAG propagation was something I'd been meaning to understand properly for a while, but honestly it wasn't the bottleneck in my setup so I'd never really dug in. Now that I actually did, a lot of things clicked into place, and I picked up a fair amount along the way — so if anything, your post gave me a good excuse to finally sit with it. Good luck with the competition!" }, "source": "meta" }