This page propose some indications for your work and how you can manage the simpsons dataset:


## 1. Import and dataset read

```python
import pandas as pd
import os
import h5py
from skimage import io, color, exposure, transform
import numpy as np
```

``` python
# read data from dataset
df = pd.read_csv('/kaggle/input/simpsons-faces/train.csv')
df.head()
```

## 2. Quick data separation

```python
data       = df.sample(frac=1., axis=0, random_state=42)
data_train = data.sample(frac=0.8, axis=0, random_state=42)
data_test  = data.drop(data_train.index)
```

## 3.  Iterate over a dataframe in order to preprocess images



```python
out_x_train = []
img_size = (64, 64)
folder = '/kaggle/input/simpsons-faces/simpsons/train'

# `df` is the current dataframe variable
for i, row in df.iterrows():
        
    img_path = os.path.join(folder, row['path'])
    img_arr = io.imread(img_path)
    
    # resize the image
    img_arr = transform.resize(img_arr, img_size, preserve_range=True)
        
    # TODO : do whatever you want in preprocessing
    # See `skimage.transform` if needed

    # normalize image
    img_arr = img_arr / 255.
        
    out_x_train.append(img_arr)
```

## 4. Create dataset h5 file

The h5 file format makes it possible to quickly load a large database where pre-processing has been performed on data 
```python
output_folder = '/kaggle/working/datasets'
os.makedirs(output_folder, exist_ok=True)
dataset_file = os.path.join(output_folder, 'dataset_gray.h5')

# `out_x_train` and `out_x_test` lists contain preprocessed images
# => image normalization
# => gray or rgb level images (or whatever you want)
with h5py.File(dataset_file, "w") as f:
        f.create_dataset("x_train", data=out_x_train)
        f.create_dataset("y_train", data=data_train.classId.to_numpy())
        
        # TODO: Do the same for test data 
        # f.create_dataset("x_test",  data=out_x_test)
        # f.create_dataset("y_test",  data=data_test.classId.to_numpy())

# read dataset h5 file
with h5py.File(dataset_file, 'r') as f:
    x_train = f['x_train'][:]
    y_train = f['y_train'][:]
    
    # TODO: Do the same for test data
    # x_test  = f['x_test'][:]
    # y_test  = f['y_test'][:]
    
print(x_train.shape)
```

## 5. Manage inbalanced data

It may be necessary to weight minority class samples (low samples number) correctly when training as opposed to majority classes:

```python
from sklearn.utils.class_weight import compute_sample_weight

# use of sample weight
samples_weight = compute_sample_weight("balanced", y_train)
    
model.fit(..., sample_weight=samples_weight)
```