In this homework, we will train a deep learning model that can classify handwritten letters and digits in Extended-MNIST (EMNIST) dataset. The EMNIST include 52 letters (capital and small letters) and 10 digits. We will use the default test dataset (emnist-byclass-test) to evaluate your model.
\n\nYou are free to use any neural network architecture, any DL library (PyTorch, TensorFlow, ...), or even any language (C, Java, JavaScript). This is just a warm-up homework so don't worry about the accuracy too much. As long as your accuracy is better than random guess, you will get the credit.\n
\n\nPlease register a Kaggle account with your student ID. Because Kaggle only accepts output results, please upload your source code to NTUT i-School.\n
\nWe've converted the EMNIST binary data to NumPy npz format. The code below shows how to load training data:
\n\nimport numpy as np\n# Load training data\ndata = np.load('emnist-byclass-train.npz')\ntrain_labels = data['training_labels']\ntrain_data = data['training_images']\n\n# Load testing data\ntest_data = np.load('emnist-byclass-test.npz')['testing_images']\n\nYou are free to download the original EMNIST from its official website. However DON'T USE ByClass Testing Images for Training!\nThe statistics of the EMNIST is shown in the table below:
\n\n\nThe By_Class split includes 62 classes comprising [0-9], [a-z] and [A-Z], while the By_Merge split has merged the uppercase and lowercase letters of C, I, J, K, L, M, O, P, S, U, V, W, X, Y and Z, and include only 47 classes.
\nA submission file of prediction results should be a CSV file containing two columns: Id and Category, separated by ','.
\nThe file should contain a header and have the following format:
\nId,Category\n0,15\n1,7\n2,30\n3,22\n...\n\n\nWe will use Category Accuracy as the metric. There are 62 classes (0 ~ 9 & A ~ Z & a ~z) in ByClass split of EMNIST dataset, which are labeled sequentially as 0, 1, 2, ..., 61.\n\n
The evaluation pages describes how submissions will be scored and how students should format their submissions. You don't need a sub-title at the top (the page title appears above). For an example of a high quality evaluation page, see the 2013 KDD Cup page, which we have copied here:
\r\nThe evaluation metric for this competition is Mean F1-Score. The F1 score, commonly used in information retrieval, measures accuracy using the statistics precision p and recall r. Precision is the ratio of true positives (tp) to all predicted positives (tp + fp). Recall is the ratio of true positives to all actual positives (tp + fn). The F1 score is given by:
\r\n\\[ F1 = 2\\frac{p \\cdot r}{p+r}\\ \\ \\mathrm{where}\\ \\ p = \\frac{tp}{tp+fp},\\ \\ r = \\frac{tp}{tp+fn} \\]
\r\nThe F1 metric weights recall and precision equally, and a good retrieval algorithm will maximize both precision and recall simultaneously. Thus, moderately good performance on both will be favored over extremely good performance on one and poor performance on the other.
\r\nFor every author in the dataset, submission files should contain two columns: AuthorId and DuplicateAuthorIds. DuplicateAuthorIds should be a space-delimited list. Every AuthorId counts as his/her own duplicate, and every duplicate should be listed under each of its respective ids. For example, if you suspect author A, B, and C are the same, you should list (A,A B C), (B,B A C), (C,C A B).
\r\nThe file should contain a header and have the following format:
\r\nAuthorId,DuplicateAuthorIds", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false } ] }
1,1
8,8
9,9 10
10,10 9
etc.