Paper: Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt (only a few weeks old)
\nTraining even 1 epoch (~150k notebooks) on the Kaggle P100 could take hours. A simple baseline to speed up training could be to remove duplicates or near-duplicate notebooks [1].
\nThis paper introduces RHO-LOSS, which lets the authors achieve their target accuracy in 18x fewer training steps on large noisy datasets like Clothing-1M.
\n\n\nTraining on web-scale data can take months. But most computation and time is wasted on redundant and noisy points that are already learnt or not learnable.
\n
They prioritize training on samples that are most likely to result in decreased validation accuracy according to a simple heuristic.
\nThe main idea is to avoid samples that are noisy, irrelevant, and redundant. The authors show that RHO-LOSS chooses low-noise, task-relevant, non-redundant samples for training.
\nIn terms of epochs of training, their RHO-LOSS sampling often yields large reductions compared to uniform sampling and other curriculum-based baselines.
\n[1] The Adverse Effects of Code Duplication in Machine Learning Models of Code
", "messages": [ { "id": 1853477, "postDate": "2022-07-12T23:05:31.757Z", "content": "Paper: Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt (only a few weeks old)
\nTraining even 1 epoch (~150k notebooks) on the Kaggle P100 could take hours. A simple baseline to speed up training could be to remove duplicates or near-duplicate notebooks [1].
\nThis paper introduces RHO-LOSS, which lets the authors achieve their target accuracy in 18x fewer training steps on large noisy datasets like Clothing-1M.
\n\n\nTraining on web-scale data can take months. But most computation and time is wasted on redundant and noisy points that are already learnt or not learnable.
\n
They prioritize training on samples that are most likely to result in decreased validation accuracy according to a simple heuristic.
\nThe main idea is to avoid samples that are noisy, irrelevant, and redundant. The authors show that RHO-LOSS chooses low-noise, task-relevant, non-redundant samples for training.
\nIn terms of epochs of training, their RHO-LOSS sampling often yields large reductions compared to uniform sampling and other curriculum-based baselines.
\n[1] The Adverse Effects of Code Duplication in Machine Learning Models of Code
", "rawMarkdown": "__Paper__: [Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt](https://arxiv.org/abs/2206.07137) (only a few weeks old)\n\nTraining even 1 epoch (~150k notebooks) on the Kaggle P100 could take hours. A simple baseline to speed up training could be to remove duplicates or near-duplicate notebooks [1]. \n\nThis paper introduces RHO-LOSS, which lets the authors achieve their target accuracy in __18x fewer training steps__ on large noisy datasets like Clothing-1M.\n\n > Training on web-scale data can take months. But most computation and time is wasted on redundant and noisy points that are already learnt or not learnable.\n\nThey prioritize training on samples that are most likely to result in decreased validation accuracy according to a simple heuristic.\n\nThe main idea is to avoid samples that are __noisy__, __irrelevant__, and __redundant__. The authors show that RHO-LOSS chooses low-noise, task-relevant, non-redundant samples for training.\n\nIn terms of epochs of training, their RHO-LOSS sampling often yields large reductions compared to uniform sampling and other curriculum-based baselines.\n\n---\n[1] [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://arxiv.org/abs/1812.06469)", "votes": 5 }, { "id": 1853566, "postDate": "2022-07-13T00:50:06.367Z", "content": "Thank you for introducing interesting loss!
\nIs it right to say something like https://github.com/oatml/rho-loss ?
Thank you for introducing interesting loss!
\nIs it right to say something like https://github.com/oatml/rho-loss ?