{
  "id": 336770,
  "title": "JuPyT5: The Key to Winning?",
  "url": "/competitions/AI4Code/discussion/336770",
  "author_name": "Sarthak Bhatt",
  "post_date": "2022-07-12T22:28:53.144000",
  "votes": 14,
  "comment_count": 1,
  "views": 0,
  "content": "<p><a href=\"https://arxiv.org/pdf/2201.12901.pdf\" target=\"_blank\">Training and Evaluating a Jupyter Notebook Data Science Assistant</a> by Microsoft on 30th Jan 2022</p>\n<p>The authors of the paper train their models on <strong>all Jupyter Notebooks from all public GitHub repositories with Jupyter Notebooks</strong> as of April 2021. This amounts to a total of 7.24 Million <em>unique</em> Jupyter Notebooks with 221 million unique cells (153 million code cells + 67 million markdown cells). This is an order of magnitude more than the data we are given in the competition.</p>\n<p>JuPyT5 is initialized from PyMT5 model and is trained on 80x 32 GB V100 GPUs. Training on such a scale would be way beyond most of our budgets, so having this model could be very useful. </p>\n<hr>\n<p><strong>Cell Infilling Pretraining</strong>: For a notebook, a target cell is selected. This cell is removed from the input and the model is asked to reconstruct the cell given the nearby cells. This is useful to our case because reconstructing an entire cell is way more difficult than ordering some given cells, so a model that can do this should be able to solve the easier task of reordering some cells. </p>\n<p><strong>PyMT5</strong>: The model is initialized from the PyMT5 model [1]. PyMT5 model is trained on 26 million Python methods and 7.7 million method-docstring pairs. PyMT5 and JuPyT5 also share 2 common authors. </p>\n<p>The original motivation for the paper is to use JuPyT5 model as a data science assistant, although I think this part is irrelevant for our use case. The more important part is that the model is trained on ~50x more Jupyter notebooks than we are given in the competition.</p>\n<hr>\n<p>[1] <a href=\"https://arxiv.org/pdf/2010.03150.pdf\" target=\"_blank\">PYMT5: multi-mode translation of natural language and Python code with transformers</a></p>",
  "messages": [
    {
      "id": 1853459,
      "postDate": "2022-07-12T22:28:53.143Z",
      "content": "<p><a href=\"https://arxiv.org/pdf/2201.12901.pdf\" target=\"_blank\">Training and Evaluating a Jupyter Notebook Data Science Assistant</a> by Microsoft on 30th Jan 2022</p>\n<p>The authors of the paper train their models on <strong>all Jupyter Notebooks from all public GitHub repositories with Jupyter Notebooks</strong> as of April 2021. This amounts to a total of 7.24 Million <em>unique</em> Jupyter Notebooks with 221 million unique cells (153 million code cells + 67 million markdown cells). This is an order of magnitude more than the data we are given in the competition.</p>\n<p>JuPyT5 is initialized from PyMT5 model and is trained on 80x 32 GB V100 GPUs. Training on such a scale would be way beyond most of our budgets, so having this model could be very useful. </p>\n<hr>\n<p><strong>Cell Infilling Pretraining</strong>: For a notebook, a target cell is selected. This cell is removed from the input and the model is asked to reconstruct the cell given the nearby cells. This is useful to our case because reconstructing an entire cell is way more difficult than ordering some given cells, so a model that can do this should be able to solve the easier task of reordering some cells. </p>\n<p><strong>PyMT5</strong>: The model is initialized from the PyMT5 model [1]. PyMT5 model is trained on 26 million Python methods and 7.7 million method-docstring pairs. PyMT5 and JuPyT5 also share 2 common authors. </p>\n<p>The original motivation for the paper is to use JuPyT5 model as a data science assistant, although I think this part is irrelevant for our use case. The more important part is that the model is trained on ~50x more Jupyter notebooks than we are given in the competition.</p>\n<hr>\n<p>[1] <a href=\"https://arxiv.org/pdf/2010.03150.pdf\" target=\"_blank\">PYMT5: multi-mode translation of natural language and Python code with transformers</a></p>",
      "rawMarkdown": "[Training and Evaluating a Jupyter Notebook Data Science Assistant](https://arxiv.org/pdf/2201.12901.pdf) by Microsoft on 30th Jan 2022\n\nThe authors of the paper train their models on __all Jupyter Notebooks from all public GitHub repositories with Jupyter Notebooks__ as of April 2021. This amounts to a total of 7.24 Million _unique_ Jupyter Notebooks with 221 million unique cells (153 million code cells + 67 million markdown cells). This is an order of magnitude more than the data we are given in the competition.\n\nJuPyT5 is initialized from PyMT5 model and is trained on 80x 32 GB V100 GPUs. Training on such a scale would be way beyond most of our budgets, so having this model could be very useful. \n\n---\n__Cell Infilling Pretraining__: For a notebook, a target cell is selected. This cell is removed from the input and the model is asked to reconstruct the cell given the nearby cells. This is useful to our case because reconstructing an entire cell is way more difficult than ordering some given cells, so a model that can do this should be able to solve the easier task of reordering some cells. \n\n__PyMT5__: The model is initialized from the PyMT5 model [1]. PyMT5 model is trained on 26 million Python methods and 7.7 million method-docstring pairs. PyMT5 and JuPyT5 also share 2 common authors. \n\nThe original motivation for the paper is to use JuPyT5 model as a data science assistant, although I think this part is irrelevant for our use case. The more important part is that the model is trained on ~50x more Jupyter notebooks than we are given in the competition.\n\n---\n[1] [PYMT5: multi-mode translation of natural language and Python code with transformers](https://arxiv.org/pdf/2010.03150.pdf)\n\n\n ",
      "votes": 14
    },
    {
      "id": 1855547,
      "postDate": "2022-07-14T17:21:04.307Z",
      "rawMarkdown": "",
      "votes": 1,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1855547,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-07-14T17:21:04.307000",
      "content": "",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1853459": "[Training and Evaluating a Jupyter Notebook Data Science Assistant](https://arxiv.org/pdf/2201.12901.pdf) by Microsoft on 30th Jan 2022\n\nThe authors of the paper train their models on __all Jupyter Notebooks from all public GitHub repositories with Jupyter Notebooks__ as of April 2021. This amounts to a total of 7.24 Million _unique_ Jupyter Notebooks with 221 million unique cells (153 million code cells + 67 million markdown cells). This is an order of magnitude more than the data we are given in the competition.\n\nJuPyT5 is initialized from PyMT5 model and is trained on 80x 32 GB V100 GPUs. Training on such a scale would be way beyond most of our budgets, so having this model could be very useful. \n\n---\n__Cell Infilling Pretraining__: For a notebook, a target cell is selected. This cell is removed from the input and the model is asked to reconstruct the cell given the nearby cells. This is useful to our case because reconstructing an entire cell is way more difficult than ordering some given cells, so a model that can do this should be able to solve the easier task of reordering some cells. \n\n__PyMT5__: The model is initialized from the PyMT5 model [1]. PyMT5 model is trained on 26 million Python methods and 7.7 million method-docstring pairs. PyMT5 and JuPyT5 also share 2 common authors. \n\nThe original motivation for the paper is to use JuPyT5 model as a data science assistant, although I think this part is irrelevant for our use case. The more important part is that the model is trained on ~50x more Jupyter notebooks than we are given in the competition.\n\n---\n[1] [PYMT5: multi-mode translation of natural language and Python code with transformers](https://arxiv.org/pdf/2010.03150.pdf)\n\n\n ",
    "1855547": ""
  }
}