The evaluation of your team's predictions accuracy is measured the percentage of correct predictions as aggregated across all test set authors across all personality axis. That is, the performance is computed as
\[
\sum_{a \in \{cEXT, cNEU, cAGR, cCON, cOPN\}} \sum_{i\in\text{Test set authors}} I(y_{ia} = \hat y_{ia})/N.
\]
Here
\[
\begin{align}
y_{ia} &\qquad \text{is the observed label for personality axis $a$ for the $i$-th author in the test set, and} \\
\hat y_{ia} &\qquad \text{is the corresponding predicted label.}
\end{align}
\]
Furthermore N is the size of the test set, and
\[ I(x) = \begin{cases} 1 & \text{if $x = TRUE$} \\ 0 & \text{if $x=FALSE$}\end{cases} \]
Submission Format
For every author in the dataset
, submission files should contain two columns: Id and value. The Id column identifies the test set author and personality axis, by means of concatenating the
#AUTHID
string and the axis label (
EXT, NEU, CON, AGR
or
OPN
) together with an underscore (
_
) in between. The value column should contain the predicted label ('y' or 'n'). For example, if for #AUTHID
1997_504851.txt
your team's prediction is that this person is shy (EXT = 'y'), neurotic (NEU = 'y'), uncooperative (AGR = 'n'), organized (CON = 'y') and unimaginative (OPN = 'n'), then the submission file should contain the rows
⋮
1997_504851.txt_EXT,y
1997_504851.txt_NEU,y
1997_504851.txt_AGR,n
1997_504851.txt_CON,y
1997_504851.txt_OPN,n
⋮
The file should contain a header and have the following format:
Id,value
1997_504851.txt_EXT,y
1997_504851.txt_NEU,y
1997_504851.txt_AGR,n
1997_504851.txt_CON,y
1997_504851.txt_OPN,n
⋮