{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Competition Metric\nThis kernel is a short script to calculate this competition's evaluation metric: https://www.kaggle.com/c/champs-scalar-coupling/overview/evaluation in R.\n\nIt is the mean of log of MAE across the coupling types, and since it is a custom metric it won't be available as a function directly but can be easily created and used for model evaluation."},{"metadata":{"_uuid":"051d70d956493feee0c6d64651c6a088724dca2a","_execution_state":"idle","trusted":true},"cell_type":"code","source":"## loading libraries\nsuppressMessages(require(data.table))\nsuppressMessages(require(Metrics))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Competition Metric: Mean of Log MAE across type\nmlmae <- function(actuals, preds, types)\n{\n    # checking if input shapes match\n    if ((length(actuals) != length(preds)) | (length(actuals) != length(types)))\n    {\n        stop(\"Input shapes do not match\")\n    }\n    \n    # initializing values\n    sum_lmae <- 0\n    n_types <- length(unique(types))\n    \n    # calculating MAE for each type\n    for (type in unique(types))\n    {\n        sum_lmae <- sum_lmae + log(mae(actuals[types == type], preds[types == type]))\n    }\n    \n    # Mean MAE\n    return (sum_lmae / n_types)\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This function can be used while evaluating models to measure model performance on the actual competition metric."},{"metadata":{"trusted":true},"cell_type":"code","source":"## examples\ndt <- fread(\"../input/train.csv\", select = c(\"id\", \"scalar_coupling_constant\", \"type\"))\ndt[, \":=\"(zeros = 0, mean = mean(scalar_coupling_constant))]\ndt <- merge(dt, dt[, .(mean_type = mean(scalar_coupling_constant), median_type = median(scalar_coupling_constant)), .(type)], by = \"type\")\nhead(dt)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## benchmarks\ncat(\"Metric with 0s: \", mlmae(dt$scalar_coupling_constant, dt$zeros, dt$type))\ncat(\"\\nMetric with global mean: \", mlmae(dt$scalar_coupling_constant, dt$mean, dt$type))\ncat(\"\\nMetric with type mean: \", mlmae(dt$scalar_coupling_constant, dt$mean_type, dt$type))\ncat(\"\\nMetric with type median: \", mlmae(dt$scalar_coupling_constant, dt$median_type, dt$type))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":1}