{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.13"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59156,"databundleVersionId":13719397,"sourceType":"competition"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# # This Python 3 environment comes with many helpful analytics libraries installed\n# # It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# # For example, here's several helpful packages to load\n\n# import numpy as np # linear algebra\n# import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# # Input data files are available in the read-only \"../input/\" directory\n# # For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n# import os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# # You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# # You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:45.232946Z","iopub.execute_input":"2025-09-23T13:13:45.233628Z","iopub.status.idle":"2025-09-23T13:13:45.240791Z","shell.execute_reply.started":"2025-09-23T13:13:45.233593Z","shell.execute_reply":"2025-09-23T13:13:45.239732Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from pathlib import Path\nimport polars as pl\nimport warnings\nwarnings.simplefilter('ignore')\n\n\nROOT = Path(\"/kaggle\")\nDATA_DIR = ROOT / \"input/MABe-mouse-behavior-detection\"","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:45.242435Z","iopub.execute_input":"2025-09-23T13:13:45.242788Z","iopub.status.idle":"2025-09-23T13:13:46.025387Z","shell.execute_reply.started":"2025-09-23T13:13:45.242754Z","shell.execute_reply":"2025-09-23T13:13:46.024221Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data Overview\nSince the body parts traced differ by lab, let's check the details.","metadata":{}},{"cell_type":"code","source":"mice_recording_setups_df = pl.read_csv(DATA_DIR / \"train.csv\")\n\nu_lab_id = mice_recording_setups_df[\"lab_id\"].unique().to_list()\nu_body_parts_traced = mice_recording_setups_df[\"body_parts_tracked\"].unique().to_list()\nu_behaviors_labeled = mice_recording_setups_df[\"behaviors_labeled\"].unique().to_list()\n\nprint(f\"{len(u_lab_id)=}\")\nprint(f\"{len(u_body_parts_traced)=}\")\nprint(f\"{len(u_behaviors_labeled)=}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:46.026410Z","iopub.execute_input":"2025-09-23T13:13:46.026702Z","iopub.status.idle":"2025-09-23T13:13:46.260420Z","shell.execute_reply.started":"2025-09-23T13:13:46.026678Z","shell.execute_reply":"2025-09-23T13:13:46.259220Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"mice_recording_setups_df.group_by(\"lab_id\").agg(\n    pl.col(\"body_parts_tracked\").n_unique(),\n    pl.col(\"behaviors_labeled\").n_unique(),\n).select(\"body_parts_tracked\", \"behaviors_labeled\").max()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:46.261616Z","iopub.execute_input":"2025-09-23T13:13:46.261974Z","iopub.status.idle":"2025-09-23T13:13:46.328033Z","shell.execute_reply.started":"2025-09-23T13:13:46.261943Z","shell.execute_reply":"2025-09-23T13:13:46.326713Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This result indicates that multiple patterns of body parts tracked and behaviors labeled exist within a single lab.","metadata":{}},{"cell_type":"markdown","source":"## Distribution of Mouse Strains Used by Each Lab","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\n\npivot = mice_recording_setups_df.pivot(\n    index=\"lab_id\",\n    on=\"mouse1_strain\",\n    values=\"mouse1_id\",\n    aggregate_function=pl.len(),\n)\n\nsns.heatmap(\n    pivot.to_pandas().set_index(\"lab_id\"),\n    cmap=\"viridis\",\n    annot=True,\n    # fmt=\"d\",\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:46.331083Z","iopub.execute_input":"2025-09-23T13:13:46.331578Z","iopub.status.idle":"2025-09-23T13:13:50.460808Z","shell.execute_reply.started":"2025-09-23T13:13:46.331538Z","shell.execute_reply":"2025-09-23T13:13:50.459523Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- `CD-1 (ICR)`, `129/SvEvTac`, `C57BI/6J x Ai148`, and `CFW` are each used in only one lab.\n- `C57BI/6N`, `BTBR`, and `CD1` are used in two labs.\n- `C57BI/6J` is used in many labs.\n\nThe largest amount of data comes from MABe22.  \nIf we train a model without considering this, there is a risk of overfitting to the MABe22 data.  \nMany strains have only a small amount of data, so if there are behavioral differences between strains, it may be worth considering collecting additional data for those strains.","metadata":{}},{"cell_type":"markdown","source":"## Distribution of Mouse Features","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\nfeatures = [\"strain\", \"color\", \"sex\", \"age\"]\nfor feature in features:\n    fig, axes = plt.subplots(1, 4, figsize=(24, 4), sharey=True, sharex=True)\n    fig.suptitle(feature)\n    for i in range(4):\n        ax = axes[i]\n        sns.histplot(\n            data=mice_recording_setups_df,\n            x=f\"mouse{i+1}_{feature}\",\n            # hue=\"lab_id\",\n            # multiple=\"stack\",\n            ax=ax,\n        )\n        # using a FixedLocator to set the tick labels\n        ax.set_xticklabels(ax.get_xticklabels(), rotation=90)\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:50.461964Z","iopub.execute_input":"2025-09-23T13:13:50.462561Z","iopub.status.idle":"2025-09-23T13:13:53.000869Z","shell.execute_reply.started":"2025-09-23T13:13:50.462523Z","shell.execute_reply":"2025-09-23T13:13:52.999685Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Most of the colors are black or black and tan. There are only a few white mice.\n- Most of the mice are male.\n- Most ages are between 10 and 20 weeks.\n\nSince sex and age are dominated by specific values, further analysis of these features may not be meaningful.\nLet's check if there is any relationship between strain and color.","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 4, figsize=(24, 4))\nfor i in range(4):\n    ax = axes[i]\n    sns.histplot(\n        data=mice_recording_setups_df,\n        x=f\"mouse{i+1}_strain\",\n        y=f\"mouse{i+1}_color\",\n        cmap=\"viridis\",\n        ax=ax,\n        cbar=True,\n    )\n    ax.set_xticklabels(ax.get_xticklabels(), rotation=90)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:53.002069Z","iopub.execute_input":"2025-09-23T13:13:53.002697Z","iopub.status.idle":"2025-09-23T13:13:53.907379Z","shell.execute_reply.started":"2025-09-23T13:13:53.002658Z","shell.execute_reply":"2025-09-23T13:13:53.906219Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- The most common strain, `C57Gl/6J`, only appears with the color `black`.\n- The next most common strain, `BTBR`, only appears with the color `black and tan`.\n- No strain has multiple colors.","metadata":{}},{"cell_type":"markdown","source":"## Differences in Experimental Conditions","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\n\npivot = mice_recording_setups_df.pivot(\n    index=\"arena_shape\",\n    on=\"arena_type\",\n    values=\"lab_id\",\n    aggregate_function=pl.len(),\n)\n\ng = sns.heatmap(\n    pivot.to_pandas().set_index(\"arena_shape\"),\n    cmap=\"viridis\",\n    annot=True,\n    # fmt=\"d\",\n)\ng.set_xlabel(\"arena_type\");","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:53.908317Z","iopub.execute_input":"2025-09-23T13:13:53.908648Z","iopub.status.idle":"2025-09-23T13:13:54.200027Z","shell.execute_reply.started":"2025-09-23T13:13:53.908621Z","shell.execute_reply":"2025-09-23T13:13:54.198852Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Most of the combinations are (square, neutral) (n = 7944).\n- There are a few (rectangular, resident-intruder) combinations (n = 692).","metadata":{}},{"cell_type":"code","source":"bpbl_pair_counts = (\n    mice_recording_setups_df\n    .filter(\n        (pl.col(\"arena_shape\") == \"square\") & (pl.col(\"arena_type\") == \"neutral\")\n        | (pl.col(\"arena_shape\") == \"rectangular\") & (pl.col(\"arena_type\") == \"resident-intruder\")\n    )\n    .group_by(\"body_parts_tracked\", \"behaviors_labeled\")\n    .len(\"count\")\n    .sort([\"count\", \"body_parts_tracked\", \"behaviors_labeled\"], descending=True)\n)\n\nbpbl_pair_counts.head(10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:54.201027Z","iopub.execute_input":"2025-09-23T13:13:54.201306Z","iopub.status.idle":"2025-09-23T13:13:54.250491Z","shell.execute_reply.started":"2025-09-23T13:13:54.201283Z","shell.execute_reply":"2025-09-23T13:13:54.249142Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"filtered = mice_recording_setups_df.filter(\n    (\n        (pl.col(\"arena_shape\") == \"square\") & (pl.col(\"arena_type\") == \"neutral\")\n        | (pl.col(\"arena_shape\") == \"rectangular\") & (pl.col(\"arena_type\") == \"resident-intruder\")\n    )\n    & (pl.col(\"behaviors_labeled\").is_null())\n)\n\nprint(f\"{filtered['lab_id'].unique().to_list()=}\")\nprint(f\"{filtered['frames_per_second'].unique().to_list()=}\")\nprint(f\"{filtered['video_duration_sec'].unique().to_list()=}\")\nprint(f\"{filtered['pix_per_cm_approx'].unique().to_list()=}\")\nprint(f\"{filtered['video_width_pix'].unique().to_list()=}\")\nprint(f\"{filtered['video_height_pix'].unique().to_list()=}\")\nprint(f\"{filtered['arena_width_cm'].unique().to_list()=}\")\nprint(f\"{filtered['arena_height_cm'].unique().to_list()=}\")\nprint(f\"{filtered['tracking_method'].unique().to_list()=}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:54.251632Z","iopub.execute_input":"2025-09-23T13:13:54.252014Z","iopub.status.idle":"2025-09-23T13:13:54.276395Z","shell.execute_reply.started":"2025-09-23T13:13:54.251978Z","shell.execute_reply":"2025-09-23T13:13:54.275318Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"mice_recording_setups_df.filter(\n    pl.col(\"lab_id\").is_in([\"MABe22_keypoints\", \"MABe22_movies\"])\n).height","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:54.277888Z","iopub.execute_input":"2025-09-23T13:13:54.278129Z","iopub.status.idle":"2025-09-23T13:13:54.310922Z","shell.execute_reply.started":"2025-09-23T13:13:54.278109Z","shell.execute_reply":"2025-09-23T13:13:54.309459Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"For MABe22_keypoints and MABe22_movies, `behaviors_labeled` is `null`.","metadata":{}},{"cell_type":"markdown","source":"## Video Features","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\nnumerical_features = [\"frames_per_second\", \"video_duration_sec\", \"pix_per_cm_approx\", \"video_width_pix\", \"video_height_pix\", \"arena_width_cm\", \"arena_height_cm\"]\nfig, axes = plt.subplots(1, len(numerical_features), figsize=(24, 4), sharey=True)\nfor i, feature in enumerate(numerical_features):\n    ax = axes[i]\n    sns.histplot(\n        data=mice_recording_setups_df,\n        x=feature,\n        # hue=\"lab_id\",\n        # multiple=\"stack\",\n        ax=ax,\n    )\n    # using a FixedLocator to set the tick labels\n    # ax.set_xticklabels(ax.get_xticklabels(), rotation=90)\n    ax.set_title(feature)\n    ax.set_yscale(\"log\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:54.312284Z","iopub.execute_input":"2025-09-23T13:13:54.312744Z","iopub.status.idle":"2025-09-23T13:13:56.752489Z","shell.execute_reply.started":"2025-09-23T13:13:54.312703Z","shell.execute_reply":"2025-09-23T13:13:56.750639Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import seaborn as sns\nimport numpy as np\n\ncorr = mice_recording_setups_df.select(numerical_features).corr().to_pandas()\ncorr.index = corr.columns\nmask = np.triu(np.ones_like(corr, dtype=bool))\n\nsns.heatmap(\n    corr,\n    cmap=\"viridis\",\n    annot=True,\n    fmt=\".2f\",\n    vmin=-1,\n    vmax=1,\n    square=True,\n    mask=mask,\n    linewidths=0.5,\n    cbar_kws={\"shrink\": 0.75},\n)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:56.753776Z","iopub.execute_input":"2025-09-23T13:13:56.754113Z","iopub.status.idle":"2025-09-23T13:13:57.217773Z","shell.execute_reply.started":"2025-09-23T13:13:56.754080Z","shell.execute_reply":"2025-09-23T13:13:57.216550Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"features = [\"frames_per_second\", \"video_duration_sec\"]\ncounts = mice_recording_setups_df.select(\n    pl.col(\"lab_id\"),\n    *[pl.col(col) for col in features],\n).unique().sort(\"lab_id\").group_by(\"lab_id\").len(\"count\")\n\ntotals = mice_recording_setups_df.group_by(\"lab_id\").len(\"total\")\n\ncounts = (\n    counts\n    .join(totals, on=\"lab_id\", how=\"left\")\n    .with_columns(\n        (pl.col(\"count\") / pl.col(\"total\")).alias(\"ratio\"),\n    )\n    .sort(\"ratio\", descending=True)\n)\ndisplay(counts.head(10))\ndisplay(counts.tail(10))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:57.220753Z","iopub.execute_input":"2025-09-23T13:13:57.221086Z","iopub.status.idle":"2025-09-23T13:13:57.270885Z","shell.execute_reply.started":"2025-09-23T13:13:57.221059Z","shell.execute_reply":"2025-09-23T13:13:57.269367Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- `CaIMS21_*` has the second largest amount of data after `MABe22_*`, but (`fps`, `video_duration_sec`) are different.\n- The largest dataset, `MABe22_*`, appears to have been recorded under completely identical conditions.","metadata":{}},{"cell_type":"code","source":"features = [\"pix_per_cm_approx\", \"video_width_pix\", \"video_height_pix\"]\ncounts = mice_recording_setups_df.select(\n    pl.col(\"lab_id\"),\n    *[pl.col(col) for col in features],\n).unique().sort(\"lab_id\").group_by(\"lab_id\").len(\"count\")\n\ntotals = mice_recording_setups_df.group_by(\"lab_id\").len(\"total\")\n\ncounts = (\n    counts\n    .join(totals, on=\"lab_id\", how=\"left\")\n    .with_columns(\n        (pl.col(\"count\") / pl.col(\"total\")).alias(\"ratio\"),\n    )\n    .sort(\"ratio\", descending=True)\n)\ndisplay(counts.head(10))\ndisplay(counts.tail(10))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:57.271873Z","iopub.execute_input":"2025-09-23T13:13:57.272175Z","iopub.status.idle":"2025-09-23T13:13:57.296050Z","shell.execute_reply.started":"2025-09-23T13:13:57.272142Z","shell.execute_reply":"2025-09-23T13:13:57.294862Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"It appears that the video settings during recording are fixed for `MABe22_*` and `CaIMS21_*`.","metadata":{}},{"cell_type":"code","source":"features = [\"arena_width_cm\", \"arena_height_cm\", \"arena_shape\", \"arena_type\"]\ncounts = mice_recording_setups_df.select(\n    pl.col(\"lab_id\"),\n    *[pl.col(col) for col in features],\n).unique().sort(\"lab_id\").group_by(\"lab_id\").len(\"count\")\n\ntotals = mice_recording_setups_df.group_by(\"lab_id\").len(\"total\")\n\ncounts = (\n    counts\n    .join(totals, on=\"lab_id\", how=\"left\")\n    .with_columns(\n        (pl.col(\"count\") / pl.col(\"total\")).alias(\"ratio\"),\n    )\n    .sort(\"ratio\", descending=True)\n)\ndisplay(counts.head(10))\ndisplay(counts.tail(10))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T13:13:57.297633Z","iopub.execute_input":"2025-09-23T13:13:57.297977Z","iopub.status.idle":"2025-09-23T13:13:57.321081Z","shell.execute_reply.started":"2025-09-23T13:13:57.297946Z","shell.execute_reply":"2025-09-23T13:13:57.319693Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"It appears that the mouse experimental environment is fixed for `MABe22_*` and `CaIMS21_*`.","metadata":{}},{"cell_type":"markdown","source":"# Literature Information","metadata":{}},{"cell_type":"markdown","source":"from [MABe22 at ICML 2023](https://arxiv.org/pdf/2207.10553.pdf)\n\n> <span style=\"color:blue\">What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.</span>\n> \n> The core element of this dataset, called a sequence, consists of raw video, tracked postures, sequence-level experimental conditions, and hand-scored actions of three mice interacting in a 52 cm x 52 cm arena, filmed from above at 30 Hz. All three mice are adult males from the same strain, either C57Bl/6J or BTBR. Postures of animals are estimated in terms of a set of twelve anatomically defined ”keypoints” that capture the detailed two-dimensional pose of the animal. Because the three mice are not easily distinguished, temporal filtering methods are used to track the identity of animals across frames. <span style=\"color:red\">Because both of these processing steps are automated, some errors in pose estimation or swaps of mouse identity do occur in the dataset. Accompanying each sequence are frame-by-frame annotations for 8 ”hidden tasks” capturing experimental conditions, animal background, and animal behavior. The 8 hidden tasks for this dataset include four ”sequence-level” tasks where annotation values are the same for all frames in a one-minute sequence, and nine ”frame-level” tasks where annotation values vary from frame to frame. Descriptions of each task are provided in Table 12; all behaviors are defined between any given pair of animals.  The core element of a sequence is called a frame; this refers to the posture of the three animals on a particular frame of video, as well as annotations for the 8 hidden tasks.</span>\n\n\n> <span style=\"color:blue\">Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).</span>\n> \n> The dataset is derived from a larger experiment, in which three mice were allowed to freely interact in an open arena for a period of four days. To generate the trajectories used for this dataset, we randomly sampled up to five one-minute intervals from each recorded hour of approximately 12 such four-day experiments. In initial sampling, we observed that during the lights-on phase of the light/dark cycle the mice spent the majority of the time huddled together sleeping. <span style=\"color:red\">As this does not generate particularly interesting behavioral data, we randomly discarded 80% of sampled one-minute intervals in which no substantial movement of the animals occurred, and replaced these with substitute samples drawn from the same one-hour time period. If after five attempts we could not randomly draw a replacement sample containing movement, we omitted the trajectory from the dataset. As a result, the dataset contains a higher proportion of trajectories with movement than is present in the source videos, and a slightly lower proportion of trajectories sampled from the light portion of the light/dark cycle.</span>\n\n\n> <span style=\"color:blue\">What data does each instance consist of? “Raw” data (e.g., unprocessed text or images) or features? In either case, please provide a description.</span>\n> \n> Each sequence has three elements. 1) Keypoints are the locations of twelve body parts on each mouse: the nose tip, left and right ears, base of neck, body centroid, base, middle, and tip of tail, and the four paws. Keypoints are estimated using a modified version of HRnet documented in (Sheppard et al., 2022). 2) Annotations are sequence-level or frame-level labels of experimental conditions or animal’s actions. Definitions of these annotations are provided in Table 12. The behavior labels were generated using a series of short scripts based on features of detected animal poses; it is therefore possible that some mis-identification of behaviors occurs. Note that this dataset does not include the original raw videos from which pose estimates were produced. This is because the objective of releasing this dataset was to determine the accuracy with which animal behavior could be detected using tracked keypoints alone.\n\n> <span style=\"color:blue\">Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.</span>\n> \n> Pose keypoints in this dataset are produced using automated pose estimation software. <span style=\"color:red\">The dataset was screened to remove sequences with poor pose estimation, detected as large jumps in the detected location of an animal, however some errors in pose estimation, missing keypoints, and noise in keypoint placement still occur.</span> These are most common on frames when the two animals are in close contact or moving very quickly. <span style=\"color:red\">Frame-by-frame annotations of behavior were generated using a series of scripts that were manually tuned by a human expert. Pose estimation errors can contribute to missed bouts or false positives for behaviors in these annotations.</span>\n\n> <span style=\"color:blue\">Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section.</span>\n> \n> No preprocessing was performed on the sequence data released in this dataset.\n\n\n> <span style=\"color:blue\">Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?  For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?</span>\n> \n> Occasional errors and identity swaps during pose estimation may impact future use of the dataset for some purposes.","metadata":{}},{"cell_type":"markdown","source":"from [CalMS21 at NeurIPS](https://arxiv.org/pdf/2104.02710.pdf)","metadata":{}},{"cell_type":"markdown","source":"> <span style=\"color:blue\">What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.</span>\n>\n> The core element of this dataset, called a sequence, captures the tracked postures and actions of two mice interacting in a standard <span style=\"color:red\">resident-intruder assay</span> filmed from above at 30Hz and manually annotated on a frameby-frame basis for one or more behaviors. <span style=\"color:red\">The resident in these assays is always a male mouse from strain C57Bl/6J, or from a transgenic line with C57Bl/6J background. The intruder is a male or female BALB/c mouse.  Resident mice may be either group-housed or single-housed, and either socially/sexually naive or experienced (all factors that impact the types of social behaviors animals show in this assay.)</span> The core element of a sequence is called a frame; this refers to the posture of both animals on a particular frame of video, as well as one or more labels indicating the type of behavior being performed on that frame (if any).  <span style=\"color:red\">The dataset is divided into four sub-sets: three collections of sequences associated with Tasks 1, 2, and 3 of the MABe Challenge, and a fourth \"Unlabeled\" collection of sequences that have only the keypoint elements with no accompanying annotations or annotator-id (see \"What data does each instance consist of?\" for explanation of these values.) Tasks 1-3 are split into train and test sets. Tasks 2 and 3 are also split by annotator-id (Task 2) or behavior (Task 3).</span>\n\n\n> <span style=\"color:blue\">Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).</span>\n>\n> The assembled dataset presented here was manually curated from a large, unreleased repository of mouse behavior videos collected across several years by multiple members of the Anderson lab. <span style=\"color:red\">Only videos of naturally occurring (not optogenetically or chemogenetically evoked) behavior were included.</span> Selection criteria are described in the \"Collection Process\" section. As a result of our selection criteria, the videos included in the Tasks 1-3 datasets may not be fully representative of mouse behavior in the resident-intruder assay: videos with minimal social interactions (when the resident ignored or avoided the intruder) were omitted in favor of including a greater number of examples of the annotated behaviors of interest.\n\n> <span style=\"color:blue\">What data does each instance consist of? “Raw” data (e.g., unprocessed text or images) or features? In either case, please provide a description.</span>\n> \n> Each sequence has three elements. 1) Keypoints are the locations of seven body parts (the nose, left and right ears, base of neck, left and right hips, and base of tail) on each of two interacting mice. Keypoints are estimated using the Mouse Action Recognition System (MARS). <span style=\"color:red\">2) Annotations are manual, frame-wise labels of an animal’s actions, for example attack, mounting, and close investigation. Depending on the behaviors annotated, only between a few percent and up to half of frames will have an annotated action; frames that do not have an annotated action are labeled as other. The other label should not be taken to indicate that no behaviors are happening, and it should not be considered a true label category for purposes of classifier performance evaluation.</span> 3) Annotator-id is a unique numeric ID indicating which (anonymized) human annotator produced the labels in Annotations. This ID is provided primarily for use in Task 2 of the MABe Challenge, which pertains to annotator style capture.  Note that this dataset does not include the original raw videos from which pose estimates were produced. This is because the objective of releasing this dataset was to determine the accuracy with which animal behavior could be detected using tracked keypoints alone.\n\n> <span style=\"color:blue\">Is there a label or target associated with each instance? If so, please provide a description.</span>\n> \n> In the Task 1, Task 2, and Task 3 datasets, the annotation field for a given behavior sequence consists of frame-wise labels of animal behaviors. <span style=\"color:red\">Note that only a minority of frames have behavior labels; remaining frames are labeled as other. Only a small number of behaviors were tracked by human annotators (most typically attack, mount, and close investigation), therefore frames labeled as other are not a homogeneous category, but may contain diverse other behaviors.</span> The \"Unlabeled\" collection of sequences has no labels, and instead contains only keypoint tracking data.\n\n> <span style=\"color:blue\">Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.</span>\n> \n> The dataset includes a recommended train/test split for Tasks 1, 2, and 3. In Tasks 2 and 3, the split was designed to provide a roughly consistent, small amount of training data for each sub-task. In Task 1, the split was manually selected so that the test set included sequences from a range of experimental conditions and dates.\n\n> <span style=\"color:blue\">Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.</span>\n> \n> Pose keypoints in this dataset are produced using automated pose estimation software (the Mouse Action Recognition System, MARS). While the entire dataset was manually screened to remove sequences with poor pose estimation, some errors in pose estimation and noise in keypoint placement still occur. These are most common on frames when the two animals are in close contact or moving very quickly. In addition, manual annotations of animal behavior are inherently subjective, and individual annotators show some variability in the precise frame-by-frame labeling of behavior sequences. An investigation of within- and between-annotator variability is included in the MARS pre-print.\n\n> <span style=\"color:blue\">Any other comments?</span>\n>\n> A subset of videos in Task 1 and the Unlabeled dataset are from animals that have been implanted with a head-mounted microendoscope or optical fiber (for fiber photometry.) Because the objective of this dataset is to learn to recognize behavior in a manner that is invariant to experimental setting, the precise preparation of the resident and intruder mice (including age, sex, past experiences, and presence of neural recording devices) is not provided in the dataset.\n\n> <span style=\"color:blue\">How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, modelbased guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.</span>\n>\n> Sequences in the dataset are derived from video of pairs of socially interacting mice engaged in a standard resident-intruder assay. <span style=\"color:red\">In this assay, a black (C57Bl/6J) male \"resident\" mouse is filmed in its home cage, and a white (BALB/c) male or female \"intruder\" mouse is manually introduced to the cage by an experimenter.</span> The animals are then allowed to freely interact for between 1-2 and 10 minutes. If there is excessive fighting (injury to either animal) the assay is stopped and that trial is discarded. Resident mice typically undergo several (3-6) resident-intruder assays per day with different intruder animals.  Poses of both mice were estimated from top-view video using MARS, and pose sequences were cropped to only include frames where both animals were present in the arena. Manual, frame-by-frame annotation of animals’ actions were performed from top- and front-view video by trained experts.\n\n> <span style=\"color:blue\">If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?</span>\n> \n> <span style=\"color:red\"> The Task 1 dataset was chosen to match the training and test sets of behavior classifiers of MARS.</span> These training and test sets, in turn, were sampled from among unpublished videos collected and annotated by a member of the Anderson lab. Selection criteria for inclusion were high annotation quality (as estimated by the individual who annotated the data) and annotation completeness; videos with diverse social behaviors (mounting and attack in addition to investigation) were favored. The Tasks 2 and 3 datasets were manually selected from among previously collected (unpublished) datasets, where selection criteria were for high annotation quality, annotation completeness, and sufficient number of behavior annotations. The Unlabeled dataset consists of videos from a subset of experiments in a recent publication[30]. The subset of experiments included in this dataset was chosen at random.\n\n> Has the dataset been used for any tasks already? If so, please provide a description.\n>\n> Yes: this dataset was released to accompany the three tasks of the 2021 Multi-Agent Behavior (MABe) Challenge, posted here. The challenge tasks are summarized as follows:\n> - Task 1, Classical Classification: train supervised classifiers to detect instances of close investigation, mounting, and attack from labeled examples. All behaviors were annotated by the same individual.\n> - Task 2, Annotation Style Transfer: given limited training examples, train classifiers to reproduce the annotation style of five additional annotators for close investigation, mounting, and attack behaviors.\n> - Task 3, Learning New Behavior: given limited training examples, train classifiers to detect instances of seven additional behaviors (names of these behaviors were anonymized for this task.)\n\n> <span style=\"color:blue\"> Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?</span>\n> \n> At time of writing there is no precise, numerical consensus definition of the mouse behaviors annotated in this dataset (and in fact even different individuals trained in the same research lab and following the same written descriptions of behavior can vary in how they define particular actions such as attack, as is evidenced in Task 2.) Future users should be aware of this limitation, and bear in mind that behavior annotations in this dataset may not always agree with the behavior annotations produced by other individuals or labs.","metadata":{}},{"cell_type":"markdown","source":"- MABe21: https://www.aicrowd.com/challenges/multi-agent-behavior-representation-modeling-measurement-and-applications  \n- MABe22: https://www.aicrowd.com/challenges/multi-agent-behavior-challenge-2022/problems/mabe-2022-mouse-triplets#public-tasks","metadata":{}}]}