Your task is to predict expression changes in scRNA-seq data induced by CRISPRi perturbations. In addition to the files which are strictly necessary for a successful submission, we provide additional files which might be useful, depending on your solution. Note that you are not restricted to these files - external data can be used as well. 

# Quickstart
 **sample_submission.csv** shows the required format.  

[This notebook](https://www.kaggle.com/code/mylliabiotechnology/echoes-of-silenced-genes-sample-submission) shows how the sample submission was created.

# What to submit
The task is to predict delta average expression (log2 fold change) of 5127 genes for each perturbation. The genes to predict are defined by the columns of **training_data_means.csv**, as well as **sample_submission.csv** (except for the `pert_symbol`/`pert_id` columns). 

In total, you will predict the effects of 120 perturbations. Sixty will be used for the leaderboard (validation set) and sixty for the final evaluation (test set). The 60 leaderboard perturbations are listed in **pert_ids_val.csv**. The `pert_id` column of your submission must contain IDs as stated in this file, i.e. in `pert_i` format. The test set of the remaining 60 perturbations for final evaluation will only be released one week before the end of the challenge. Nevertheless, the submission format is constant, requiring dummy values for the as-yet-unknown test perturbations. In other words, **all submissions must have exactly 120 rows**, with values of `pert_1`...`pert_120` in the `pert_id` column. Only the first 60 rows, `pert_1`...`pert_60`, are used for the leaderboard. Until the final test set of perturbations is released, the actual values in rows `pert_61`...`pert_120` play no role.

# Where to find the dataset in Kaggle notebooks
In case you want to use a Kaggle notebook and don't see the dataset under "competition datasets" (as reported by users), it can be additionally found in "datasets". The name is "Myllia Echoes of silenced genes competition data". https://www.kaggle.com/datasets/mylliabiotechnology/myllia-echoes-of-silenced-genes-competition-data

# Main files
**training_data_means.csv**  
Average expression values of unperturbed cells (non-targeting sgRNA) and cells with 80 selected perturbations. These cells are from the same experiment as the cells in the validation and test datasets. The last row, containing `non-targeting` in the `pert_symbol` column, is the baseline expression used for reference. **All submissions should contain delta expression relative to this row**.  
See section "Normalization Details" for the exact process by which this file was generated from raw UMI counts.  

**Columns**  
`pert_symbol` - gene symbol of the perturbation  
`A1BG, A1CF, AADAC, ..., ZWINT, ZYX` - average expression values of the 5127 genes relevant to this challenge.  

**pert_ids_val.csv**  
This file provides the link between the required perturbation ID and the actual gene symbol for the 60 leaderboard (validation set) perturbations. One week before the end of the challenge, we will release another file, **pert_ids_test.csv**, containing the ID mapping for the perturbations used in the final evaluation.

 **Columns**  
`pert` - gene symbol of the perturbation  
`class` - type of dataset (`val` or `test`; currently only `val` perturbations are released)  
`pert_id` - perturbation ID to be used in submissions

# Supplementary files
**training_cells.h5ad**, **training_cells.RDS** - single-cell resolution raw data used to create **training_data_means.csv**, provided in Scanpy h5ad and Seurat formats. These files contain raw UMI counts for 19,226 genes, as well as batch information (in the `channel` column). See section "Normalization Details" for how exactly the raw UMI counts were processed to create **training_data_means.csv**. 

**training_data_ground_truth_table.csv** - this file contains all the inputs needed to run the actual scoring function on the 80 perturbations from the training dataset. Specifically, it contains delta values, gene weights, and baseline WMAE scores. It can be directly used as input for 
[the scoring function](https://www.kaggle.com/code/mylliabiotechnology/relative-wmae-multiplied-by-wcosine).

# Normalization details
We start from UMI counts of 19,226 genes. The gene list is defined by Gencode v.46.  
Raw counts are first log-normalized: 
* Divide the UMI counts in each cell by the total UMI count in that cell
* Multiply by 10,000
* Log-transform with log1p (log(x + 1)); logarithm base = 2. 

Normalization is performed on **all 19,226 genes** to facilitate compatibility with other pre-processed datasets. The normalized data are then subset to the 5,127 genes relevant for the challenge. Finally, expression values for each gene are averaged per perturbation. We use a simple arithmetic mean, disregarding batch information.