# **Submission Guidelines and Scoring Details**

In the **Customer Flags Prediction Challenge**, your submissions will be evaluated based on how accurately you predict the four binary customer flags. This document provides detailed instructions on how your submissions will be scored and how you should format your submission files.

---

## **Scoring Methodology**

### **Evaluation Metric: Macro Averaged F1 Score**

Your model's performance will be assessed using the **Macro Averaged F1 Score**, a metric suitable for multi-label classification tasks, especially when dealing with imbalanced datasets.

#### **What is the Macro F1 Score?**

- **F1 Score**: The F1 score is the harmonic mean of precision and recall for a single class.
  
  \[
  \text{F1 Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}
  \]

- **Macro Averaging**: In macro averaging, the F1 score is calculated independently for each class (in this case, each flag) and then averaged.

  \[
  \text{Macro F1 Score} = \frac{1}{N} \sum_{i=1}^{N} \text{F1 Score}_i
  \]

  where \( N \) is the number of classes (4 in this competition).

#### **Why Use Macro F1 Score?**

- **Class Balance**: It treats all classes equally, which is important when the classes are imbalanced.
- **Comprehensive Evaluation**: Considers both false positives and false negatives.

#### **Scoring Procedure**

1. **Per-Flag Evaluation**: For each of the four flags (`seeder`, `virtual_flag`, `system_flag`, `cloned_flag`), calculate the F1 score separately.
2. **Average Calculation**: Compute the average of these four F1 scores to get the Macro F1 Score.
3. **Final Score**: The final score used for ranking on the leaderboard is the Macro F1 Score.

#### **Example Calculation**

Suppose your model achieves the following F1 scores:

- F1 Score for `seeder`: 0.80
- F1 Score for `virtual_flag`: 0.75
- F1 Score for `system_flag`: 0.85
- F1 Score for `cloned_flag`: 0.65

Then, the Macro F1 Score is:

\[
\text{Macro F1 Score} = \frac{0.80 + 0.75 + 0.85 + 0.65}{4} = \frac{3.05}{4} = 0.7625
\]

---

## **Submission Format**

Your submission should be a CSV file named `submission.csv` containing your predictions for each customer in the test set. The file must adhere to the following specifications:

### **File Format**

- **File Type**: CSV (Comma-Separated Values)
- **Encoding**: UTF-8
- **Column Headers**: Include a header row with the following column names:
  - `customer_id`
  - `seeder`
  - `virtual_flag`
  - `system_flag`
  - `cloned_flag`

### **Columns Description**

1. **customer_id**
   - **Type**: String
   - **Description**: The unique identifier for each customer.
   - **Requirement**: Must match the `customer_id`s provided in the test set exactly.

2. **seeder**
   - **Type**: Integer (0 or 1)
   - **Description**: Your prediction for the `seeder` flag.
   - **Values**: 
     - `1` if you predict the customer is a seeder.
     - `0` if you predict the customer is not a seeder.

3. **virtual_flag**
   - **Type**: Integer (0 or 1)
   - **Description**: Your prediction for the `virtual_flag`.
   - **Values**:
     - `1` if you predict the customer is a virtual account.
     - `0` if you predict the customer is a real account.

4. **system_flag**
   - **Type**: Integer (0 or 1)
   - **Description**: Your prediction for the `system_flag`.
   - **Values**:
     - `1` if you predict the customer is a system-generated account.
     - `0` if you predict the customer is not a system-generated account.

5. **cloned_flag**
   - **Type**: Integer (0 or 1)
   - **Description**: Your prediction for the `cloned_flag`.
   - **Values**:
     - `1` if you predict the customer account is cloned.
     - `0` if you predict the customer account is not cloned.

### **Formatting Requirements**

- **Order of Rows**: The `customer_id`s in your submission must correspond exactly to those in the test set. The order of the rows does not matter, but each `customer_id` must be present exactly once.
- **No Missing Values**: Ensure there are no missing values in any of the columns.
- **Data Types**: Predictions must be integers (`0` or `1`), not strings or floats.
- **No Additional Columns**: Do not include any extra columns besides the ones specified.
- **No Index Column**: If exporting from pandas or similar tools, ensure that the index is not included as an extra column.

### **Sample Submission File**

Here is an example of how your `submission.csv` file should look:

```csv
customer_id,seeder,virtual_flag,system_flag,cloned_flag
1,0,1,0,0
2,1,0,0,1
3,0,0,0,0
4,1,0,1,0
5,0,1,0,0
...
```

### **Common Pitfalls to Avoid**

- **Incorrect `customer_id`s**: Make sure that all `customer_id`s in your submission match those in the test set and that there are no duplicates or missing IDs.
- **Data Type Errors**: Ensure that your predictions are integers (`0` or `1`), not floats (e.g., `0.0`, `1.0`) or strings (e.g., `"0"`, `"1"`).
- **Extra Columns or Rows**: Do not include any additional columns or rows (e.g., a header row for the index).

---

## **Submission Procedure**

1. **Generate Predictions**

   - Use your trained model(s) to predict the flags for each customer in the test set.
   - Ensure that the predictions are in the correct format.

2. **Create Submission File**

   - Compile your predictions into a CSV file named `submission.csv`.
   - Verify that the file meets all formatting requirements.

3. **Validate Submission File**

   - Open the CSV file in a text editor or spreadsheet software to confirm the structure.
   - Check for any formatting issues, such as unintended quotation marks or extra delimiters.

4. **Submit on Kaggle**

   - Log in to your Kaggle account.
   - Navigate to the competition's **"Submit Predictions"** page.
   - Upload your `submission.csv` file.
   - Confirm the submission.

5. **Check Leaderboard**

   - After submitting, your submission will be scored based on the evaluation metric.
   - Your score and ranking will appear on the public leaderboard.

---

## **Submission Limits and Rules**

- **Daily Submission Limit**: You can submit up to **5** times per day.
- **Total Submission Limit**: There is a maximum total of **[Total Submissions Allowed]** submissions throughout the competition.
- **Final Submission**: Only your **last** submission before the competition deadline will be considered for the final evaluation.

### **Leaderboard**

- **Public Leaderboard**: Displays scores based on a portion of the test set.
- **Private Leaderboard**: Final rankings are determined using the remaining portion of the test set after the competition ends.

---

## **Tips for Successful Submissions**

- **Consistency**: Double-check that the `customer_id`s in your submission align perfectly with those in the test set.
- **Version Control**: Keep track of different versions of your models and submission files to understand what works best.
- **Validation**: Use cross-validation and a hold-out validation set to estimate your model's performance before submitting.
- **Efficiency**: Given the submission limits, prioritize submitting your best models and avoid using all submissions early in the day.

---

## **Troubleshooting Submission Errors**

- **Invalid File Format**: Ensure your file is saved as a CSV with UTF-8 encoding.
- **Incorrect Column Names**: Column headers must match exactly (`customer_id`, `seeder`, `virtual_flag`, `system_flag`, `cloned_flag`).
- **Missing Predictions**: All `customer_id`s from the test set must be included with predictions for each flag.
- **Data Type Issues**: Predictions must be integers (`0` or `1`), not floats or strings.

---

## **Example Code for Creating Submission File**

Here's an example in Python using pandas to help you format your submission correctly:

```python
import pandas as pd

# Load test data
test_df = pd.read_csv('test_customer_features.csv')

# Assume you have a trained model called 'model'
# And a function 'predict_flags' that returns predictions for the four flags

# Generate predictions
# This should be replaced with your actual prediction code
predictions = predict_flags(test_df)

# Create a submission DataFrame
submission_df = pd.DataFrame({
    'customer_id': test_df['customer_id'],
    'seeder': predictions['seeder'],
    'virtual_flag': predictions['virtual_flag'],
    'system_flag': predictions['system_flag'],
    'cloned_flag': predictions['cloned_flag']
})

# Ensure predictions are integers
submission_df['seeder'] = submission_df['seeder'].astype(int)
submission_df['virtual_flag'] = submission_df['virtual_flag'].astype(int)
submission_df['system_flag'] = submission_df['system_flag'].astype(int)
submission_df['cloned_flag'] = submission_df['cloned_flag'].astype(int)

# Save to CSV
submission_df.to_csv('submission.csv', index=False)
```

---

## **Final Reminders**

- **Deadline**: Make sure to submit your final predictions before the competition deadline.
- **Compliance**: Ensure your submission complies with all competition rules and guidelines.
- **Best Practices**: Keep your code and models well-documented for potential review and learning.

---

By carefully following these guidelines, you'll help ensure that your submissions are successfully processed and fairly evaluated. Good luck with your modeling, and we look forward to seeing your innovative solutions in the **Customer Flags Prediction Challenge**!

---

If you have any questions or encounter issues with your submission, please refer to the **FAQs** or post in the competition's **Discussion Forum** for assistance.