The Mouse Action Recognition System (MARS) software pipeline for automated analysis of social behaviors in mice
\nCitation: Cristina SegalinJalani WilliamsTomomi KarigoMay HuiMoriel ZelikowskyJennifer J SunPietro PeronaDavid J AndersonAnn Kennedy (2021) The Mouse Action Recognition System (MARS) software pipeline for automated analysis of social behaviors in mice eLife 10:e63720. https://doi.org/10.7554/eLife.63720
\n\"The study of naturalistic social behavior requires quantification of animals’ interactions. This is generally done through manual annotation a highly time-consuming and tedious process. Recent advances in computer vision enable tracking the pose (posture) of freely behaving animals. However, automatically and accurately classifying complex social behaviors remains technically challenging.\"
\n\"The authors introduced the Mouse Action Recognition System (MARS), an automated pipeline for pose estimation and behavior quantification in pairs of freely interacting mice. They compared MARS’s annotations to human annotations and find that MARS’s pose estimation and behavior classification achieve human-level performance.\"
\n\"The authors also released the pose and annotation datasets used to train MARS to serve as community benchmarks and resources. Finally, they introduced the Behavior Ensemble and Neural Trajectory Observatory (BENTO), a graphical user interface for analysis of multimodal neuroscience datasets. Together, MARS and BENTO provide an end-to-end pipeline for behavior data extraction and analysis in a package that is user-friendly and easily modifiable.\"
\n\"In preliminary exploration, the authors found that high precision and recall values for individual binary behavior classifiers were achieved by gradient boosting using the XGBoost algorithm. Custom Python code to train novel behavior classifiers is included with the MARS_Developer software. Classifier hyperparameters may be set by the user, otherwise MARS will provide default values.\"
\n\"Each trained classifier produces a predicted probability that the behavior occurred, as well as a binarized output created by thresholding that probability value. Following predictions by individual classifiers, MARS combines all classifier outputs to produce a single, multi-class label for each frame of a behavior video. To do so, the authors selected on each frame the behavior label that has the highest predicted probability of occurring; if no behavior has a predicted probability of >0.5, then the frame is labeled as ‘other’ (no behavior occurring).\"
\n\"The advantage of this approach over training multi-class XGBoost is that it allows their ensemble of classifiers to be more easily expanded in the future to include additional behaviors of interest because it does not require the original training set to be fully re-annotated for the new behavior.\"
\nhttps://elifesciences.org/articles/63720
", "messages": [ { "id": "3295172", "postDate": "09/27/2025 23:37:56", "content": "The Mouse Action Recognition System (MARS) software pipeline for automated analysis of social behaviors in mice
\nCitation: Cristina SegalinJalani WilliamsTomomi KarigoMay HuiMoriel ZelikowskyJennifer J SunPietro PeronaDavid J AndersonAnn Kennedy (2021) The Mouse Action Recognition System (MARS) software pipeline for automated analysis of social behaviors in mice eLife 10:e63720. https://doi.org/10.7554/eLife.63720
\n\"The study of naturalistic social behavior requires quantification of animals’ interactions. This is generally done through manual annotation a highly time-consuming and tedious process. Recent advances in computer vision enable tracking the pose (posture) of freely behaving animals. However, automatically and accurately classifying complex social behaviors remains technically challenging.\"
\n\"The authors introduced the Mouse Action Recognition System (MARS), an automated pipeline for pose estimation and behavior quantification in pairs of freely interacting mice. They compared MARS’s annotations to human annotations and find that MARS’s pose estimation and behavior classification achieve human-level performance.\"
\n\"The authors also released the pose and annotation datasets used to train MARS to serve as community benchmarks and resources. Finally, they introduced the Behavior Ensemble and Neural Trajectory Observatory (BENTO), a graphical user interface for analysis of multimodal neuroscience datasets. Together, MARS and BENTO provide an end-to-end pipeline for behavior data extraction and analysis in a package that is user-friendly and easily modifiable.\"
\n\"In preliminary exploration, the authors found that high precision and recall values for individual binary behavior classifiers were achieved by gradient boosting using the XGBoost algorithm. Custom Python code to train novel behavior classifiers is included with the MARS_Developer software. Classifier hyperparameters may be set by the user, otherwise MARS will provide default values.\"
\n\"Each trained classifier produces a predicted probability that the behavior occurred, as well as a binarized output created by thresholding that probability value. Following predictions by individual classifiers, MARS combines all classifier outputs to produce a single, multi-class label for each frame of a behavior video. To do so, the authors selected on each frame the behavior label that has the highest predicted probability of occurring; if no behavior has a predicted probability of >0.5, then the frame is labeled as ‘other’ (no behavior occurring).\"
\n\"The advantage of this approach over training multi-class XGBoost is that it allows their ensemble of classifiers to be more easily expanded in the future to include additional behaviors of interest because it does not require the original training set to be fully re-annotated for the new behavior.\"
\nhttps://elifesciences.org/articles/63720
", "rawMarkdown": "## Mouse Action Recognition System (MARS)\n\nThe Mouse Action Recognition System (MARS) software pipeline for automated analysis of social behaviors in mice\n\n**Citation**: Cristina SegalinJalani WilliamsTomomi KarigoMay HuiMoriel ZelikowskyJennifer J SunPietro PeronaDavid J AndersonAnn Kennedy (2021) The Mouse Action Recognition System (MARS) software pipeline for automated analysis of social behaviors in mice eLife 10:e63720. https://doi.org/10.7554/eLife.63720\n\n\"The study of naturalistic social behavior requires quantification of animals’ interactions. This is generally done through manual annotation a highly time-consuming and tedious process. Recent advances in computer vision enable tracking the pose (posture) of freely behaving animals. However, automatically and accurately classifying complex social behaviors remains technically challenging.\"\n\n\"The authors introduced the **Mouse Action Recognition System (MARS)**, an automated pipeline for pose estimation and behavior quantification in pairs of freely interacting mice. They compared MARS’s annotations to human annotations and find that MARS’s pose estimation and behavior classification achieve human-level performance.\"\n\n### Behavior Ensemble and Neural Trajectory Observatory (BENTO)\n\n\"The authors also released the pose and annotation datasets used to train MARS to serve as community benchmarks and resources. Finally, they introduced the **Behavior Ensemble and Neural Trajectory Observatory (BENTO)**, a graphical user interface for analysis of multimodal neuroscience datasets. Together, MARS and BENTO provide an end-to-end pipeline for behavior data extraction and analysis in a package that is user-friendly and easily modifiable.\"\n\n\"In preliminary exploration, the authors found that high precision and recall values for individual binary behavior classifiers were achieved by gradient boosting using the XGBoost algorithm. Custom Python code to train novel behavior classifiers is included with the MARS_Developer software. Classifier hyperparameters may be set by the user, otherwise MARS will provide default values.\"\n\n\"Each trained classifier produces a predicted probability that the behavior occurred, as well as a binarized output created by thresholding that probability value. Following predictions by individual classifiers, MARS combines all classifier outputs to produce a single, multi-class label for each frame of a behavior video. To do so, the authors selected on each frame the behavior label that has the highest predicted probability of occurring; if no behavior has a predicted probability of >0.5, then the frame is labeled as ‘other’ (no behavior occurring).\"\n\n\"The advantage of this approach over training multi-class XGBoost is that it allows their ensemble of classifiers to be more easily expanded in the future to include additional behaviors of interest because it does not require the original training set to be fully re-annotated for the new behavior.\"\n\nhttps://elifesciences.org/articles/63720", "votes": null }, { "id": "3295866", "postDate": "09/29/2025 17:32:14", "content": "You might also enjoy the papers on the CalMS21 and MABe22 datasets that are included in this year's training dataset. The papers include descriptions of the previous years' winning solutions to related challenges:
\nCalMS21 (single-lab supervised classification): https://pmc.ncbi.nlm.nih.gov/articles/PMC11067713/
\nMABe22 (single-lab unsupervised representation learning): https://proceedings.mlr.press/v202/sun23g.html
Awesome, \"The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions\" and \"The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions\"
\nThank you Ann.
You might also enjoy the papers on the CalMS21 and MABe22 datasets that are included in this year's training dataset. The papers include descriptions of the previous years' winning solutions to related challenges:
\nCalMS21 (single-lab supervised classification): https://pmc.ncbi.nlm.nih.gov/articles/PMC11067713/
\nMABe22 (single-lab unsupervised representation learning): https://proceedings.mlr.press/v202/sun23g.html
Awesome, \"The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions\" and \"The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions\"
\nThank you Ann.