{
  "id": 458862,
  "title": "Aggregate the dataframe based on 'series_id' and 'event': Post processing",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/458862",
  "author_name": "",
  "post_date": "2023-12-02T00:34:22.422961400Z",
  "votes": 7,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Aggregate the dataframe based on 'series_id' and 'event', calculating the average of 'step' and 'score' for data points within the step range in each group. The 'step' is converted to int64.  Also, adds a unique 'row_id' to each row in the final DataFrame after aggregation.</p>\n<p>You can also choose how wide you want to adjust. The same events within the width will have their scores averaged.</p>\n<pre><code> pandas  pd\n\n ():\n    \n\n    aggregated_data = []\n\n    \n    sorted_df = df.sort_values(by=[, ])\n\n     (series_id, event), group  sorted_df.groupby([, ]):\n        start_step = group[].iloc[]\n        current_segment = []\n\n         _, row  group.iterrows():\n             row[] - start_step &lt;= step_range:\n                current_segment.append(row)\n            :\n                 current_segment:\n                    segment_df = pd.DataFrame(current_segment)\n                    average_step = segment_df[].mean()\n                    average_score = segment_df[].mean()\n                    aggregated_data.append({: series_id, : event, : average_step, : average_score})\n                start_step = row[]\n                current_segment = [row]\n\n        \n         current_segment:\n            segment_df = pd.DataFrame(current_segment)\n            average_step = segment_df[].mean()\n            average_score = segment_df[].mean()\n            aggregated_data.append({: series_id, : event, : average_step, : average_score})\n\n    \n    final_df = pd.DataFrame(aggregated_data)\n    final_df[] = final_df[].astype()\n    final_df = final_df.sort_values(by=[, ]).reset_index(drop=)\n    final_df.reset_index(inplace=)\n    final_df.rename(columns={: }, inplace=)\n\n    \n    final_df = final_df[[, , , , ]]\n\n     final_df\n\n\nsegments_with_modified_score = aggregate_data_by_series_event(sub_df2, )\n</code></pre>",
  "messages": [
    {
      "id": "2545948",
      "postDate": "12/02/2023 00:34:22",
      "content": "<p>Aggregate the dataframe based on 'series_id' and 'event', calculating the average of 'step' and 'score' for data points within the step range in each group. The 'step' is converted to int64.  Also, adds a unique 'row_id' to each row in the final DataFrame after aggregation.</p>\n<p>You can also choose how wide you want to adjust. The same events within the width will have their scores averaged.</p>\n<pre><code> pandas  pd\n\n ():\n    \n\n    aggregated_data = []\n\n    \n    sorted_df = df.sort_values(by=[, ])\n\n     (series_id, event), group  sorted_df.groupby([, ]):\n        start_step = group[].iloc[]\n        current_segment = []\n\n         _, row  group.iterrows():\n             row[] - start_step &lt;= step_range:\n                current_segment.append(row)\n            :\n                 current_segment:\n                    segment_df = pd.DataFrame(current_segment)\n                    average_step = segment_df[].mean()\n                    average_score = segment_df[].mean()\n                    aggregated_data.append({: series_id, : event, : average_step, : average_score})\n                start_step = row[]\n                current_segment = [row]\n\n        \n         current_segment:\n            segment_df = pd.DataFrame(current_segment)\n            average_step = segment_df[].mean()\n            average_score = segment_df[].mean()\n            aggregated_data.append({: series_id, : event, : average_step, : average_score})\n\n    \n    final_df = pd.DataFrame(aggregated_data)\n    final_df[] = final_df[].astype()\n    final_df = final_df.sort_values(by=[, ]).reset_index(drop=)\n    final_df.reset_index(inplace=)\n    final_df.rename(columns={: }, inplace=)\n\n    \n    final_df = final_df[[, , , , ]]\n\n     final_df\n\n\nsegments_with_modified_score = aggregate_data_by_series_event(sub_df2, )\n</code></pre>",
      "rawMarkdown": "Aggregate the dataframe based on 'series_id' and 'event', calculating the average of 'step' and 'score' for data points within the step range in each group. The 'step' is converted to int64.  Also, adds a unique 'row_id' to each row in the final DataFrame after aggregation.\n\nYou can also choose how wide you want to adjust. The same events within the width will have their scores averaged.\n\n```\nimport pandas as pd\n\ndef aggregate_data_by_series_event(df, step_range):\n    \"\"\"\n    :param df: DataFrame to aggregate.\n    :param step_range: The range within which steps are considered close.\n    :return: A single DataFrame with aggregated rows.\n    \"\"\"\n\n    aggregated_data = []\n\n    # Sorting the DataFrame by 'series_id' and 'step' \n    sorted_df = df.sort_values(by=['series_id', 'step'])\n\n    for (series_id, event), group in sorted_df.groupby(['series_id', 'event']):\n        start_step = group['step'].iloc[0]\n        current_segment = []\n\n        for _, row in group.iterrows():\n            if row['step'] - start_step <= step_range:\n                current_segment.append(row)\n            else:\n                if current_segment:\n                    segment_df = pd.DataFrame(current_segment)\n                    average_step = segment_df['step'].mean()\n                    average_score = segment_df['score'].mean()\n                    aggregated_data.append({'series_id': series_id, 'event': event, 'step': average_step, 'score': average_score})\n                start_step = row['step']\n                current_segment = [row]\n\n        # Aggregating the last segment for each series_id and event\n        if current_segment:\n            segment_df = pd.DataFrame(current_segment)\n            average_step = segment_df['step'].mean()\n            average_score = segment_df['score'].mean()\n            aggregated_data.append({'series_id': series_id, 'event': event, 'step': average_step, 'score': average_score})\n\n    # Creating the final DataFrame, converting 'step' to int64, and adding 'row_id'\n    final_df = pd.DataFrame(aggregated_data)\n    final_df['step'] = final_df['step'].astype('int64')\n    final_df = final_df.sort_values(by=['series_id', 'step']).reset_index(drop=True)\n    final_df.reset_index(inplace=True)\n    final_df.rename(columns={'index': 'row_id'}, inplace=True)\n\n    # Reordering the columns to match the desired structure\n    final_df = final_df[['row_id', 'series_id', 'step', 'event', 'score']]\n\n    return final_df\n\n# Re-segmenting the data with the new function that modifies the score\nsegments_with_modified_score = aggregate_data_by_series_event(sub_df2, 15)\n```",
      "votes": null
    },
    {
      "id": "2546159",
      "postDate": "12/02/2023 06:54:21",
      "content": "<blockquote>\n  <p>The range within which steps are considered close<br>\n  <a href=\"https://www.kaggle.com/dolphin15\" target=\"_blank\">@dolphin15</a> hey can you share which range works for you?</p>\n</blockquote>",
      "rawMarkdown": ">The range within which steps are considered close\n@dolphin15 hey can you share which range works for you?",
      "votes": null
    },
    {
      "id": "2546664",
      "postDate": "12/02/2023 17:11:26",
      "content": "<p>for me, 12 does not change my score, 360 is too big</p>",
      "rawMarkdown": "for me, 12 does not change my score, 360 is too big",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2546159,
      "author_name": "vipin20",
      "author_url": "",
      "post_date": "12/02/2023 06:54:21",
      "content": "<blockquote>\n  <p>The range within which steps are considered close<br>\n  <a href=\"https://www.kaggle.com/dolphin15\" target=\"_blank\">@dolphin15</a> hey can you share which range works for you?</p>\n</blockquote>",
      "votes": null,
      "replies": [
        {
          "id": 2546664,
          "author_name": "gentlezdh",
          "author_url": "",
          "post_date": "12/02/2023 17:11:26",
          "content": "<p>for me, 12 does not change my score, 360 is too big</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2545948": "Aggregate the dataframe based on 'series_id' and 'event', calculating the average of 'step' and 'score' for data points within the step range in each group. The 'step' is converted to int64.  Also, adds a unique 'row_id' to each row in the final DataFrame after aggregation.\n\nYou can also choose how wide you want to adjust. The same events within the width will have their scores averaged.\n\n```\nimport pandas as pd\n\ndef aggregate_data_by_series_event(df, step_range):\n    \"\"\"\n    :param df: DataFrame to aggregate.\n    :param step_range: The range within which steps are considered close.\n    :return: A single DataFrame with aggregated rows.\n    \"\"\"\n\n    aggregated_data = []\n\n    # Sorting the DataFrame by 'series_id' and 'step' \n    sorted_df = df.sort_values(by=['series_id', 'step'])\n\n    for (series_id, event), group in sorted_df.groupby(['series_id', 'event']):\n        start_step = group['step'].iloc[0]\n        current_segment = []\n\n        for _, row in group.iterrows():\n            if row['step'] - start_step <= step_range:\n                current_segment.append(row)\n            else:\n                if current_segment:\n                    segment_df = pd.DataFrame(current_segment)\n                    average_step = segment_df['step'].mean()\n                    average_score = segment_df['score'].mean()\n                    aggregated_data.append({'series_id': series_id, 'event': event, 'step': average_step, 'score': average_score})\n                start_step = row['step']\n                current_segment = [row]\n\n        # Aggregating the last segment for each series_id and event\n        if current_segment:\n            segment_df = pd.DataFrame(current_segment)\n            average_step = segment_df['step'].mean()\n            average_score = segment_df['score'].mean()\n            aggregated_data.append({'series_id': series_id, 'event': event, 'step': average_step, 'score': average_score})\n\n    # Creating the final DataFrame, converting 'step' to int64, and adding 'row_id'\n    final_df = pd.DataFrame(aggregated_data)\n    final_df['step'] = final_df['step'].astype('int64')\n    final_df = final_df.sort_values(by=['series_id', 'step']).reset_index(drop=True)\n    final_df.reset_index(inplace=True)\n    final_df.rename(columns={'index': 'row_id'}, inplace=True)\n\n    # Reordering the columns to match the desired structure\n    final_df = final_df[['row_id', 'series_id', 'step', 'event', 'score']]\n\n    return final_df\n\n# Re-segmenting the data with the new function that modifies the score\nsegments_with_modified_score = aggregate_data_by_series_event(sub_df2, 15)\n```",
    "2546159": ">The range within which steps are considered close\n@dolphin15 hey can you share which range works for you?",
    "2546664": "for me, 12 does not change my score, 360 is too big"
  },
  "source": "meta"
}