{
  "fetched_at": "2026-10-19T19:19:30Z",
  "source_url": "https://www.kaggle.com/competitions/cs446-fa19/data",
  "pages": [
    {
      "name": "rules",
      "content": "This is a page where you can include rules that participants must accept before joining. You may wish to include rules like:\n\n - Don't cheat!\n - Apply yourself!\n - Have fun!",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    },
    {
      "name": "data-description",
      "content": "# Data\nThe dataset includes a collection of textual descriptions of neural network model architectures trained on Cifar-10, i.e., inputs “X”. The labels “Y” are the final training and testing performance scores of these models. Do not train your own models on cifar-10, you are provided with training and test examples. Your goal will be to build a performance predictor for this data set.\n\n## Provided Features\nThe performance of a neural network often depends on a few factors [4,5], some of which we have selected as features:\n1.) Architecture description as a string and its hyperparameters\n2.) The first X epochs of training and validation error history\n3.) Initialization statistics. Specifically mean, std and L2 norm of the network for each layer before training starts\n\nWhile your goal is to predict the final train error and final test error, we have interleaved (and identified) examples corresponding to training and testing error. You may consider training a separate performance prediction for the train and test error.\n\n## Provided files\n- `train.csv`\n- `test.csv`\n- `sample_submission.csv`\n\n## Descriptions of files\n#### `train.csv`\nBelow I have briefly described what each column represents.\n\n- id - identification for data sample\n- arch_and_hp - model architecture and hyperparameters used with that architecture.\n- batch_size_test - number of samples that are evaluated at once for test data\n- batch_size_val - number of samples that are evaluated at once for validation data\n- criterion - loss function\n- epochs - number of epochs the model was trained \n- number_parameters - number of parameters in the model\n- optimizer - optimizer used to update gradients\n- val_error - final validation error (what you need to predict)\n- val_loss - final validation loss\n- train_error - final training error (what you need to predict)\n- train_loss - final training loss\n- batch_size_train - number of samples that are evaluated at once for training data\n- init_params_mu - mean of initial parameters\n- init_params_std - standard deviation of initial parameters\n- init_params_l2 - L2 norm of initial parameters\n- val_accs_{0,..,49} - validation accuracy for the first 50 epochs\n- val_losses_{0,..,49} - validation losses for the first 50 epochs\n- train_accs_{0,..,49} - training accuracy for the first 50 epochs\n- train_losses_{0,..,49} - training losses for the first 50 epochs\n\n#### `test.csv`\nSee description of `train.csv`. This is the same except final validation and training loss and error have been removed.\n\n#### `sample_submission.csv`\nThis file gives the desired format for your results to submit.",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    },
    {
      "name": "Description",
      "content": "# Overview\nAs you have learned in class, selecting model architectures and hyperparameters is often a difficult task in applied machine learning. Automated machine learning (AutoML) is a framework of tools designed to automate this model selection process. There is rising evidence in [1,2,3] that performance predictors have the potential to be useful in this process. Performance predictors take a dataset and hyperparameters as input and predict the performance of the resulting trained model. Thus, an effective performance predictor can be used to select good model architectures and hyperparameters, e.g., since the learned predictor is a function, one could create approaches for optimizing it. In this project, your goal will be to train a performance predictor.\n\nEach example in the training set is a network architecture, associated meta-data (details below), and the train and test performance of the associated model when it was trained to convergence. Your task is to develop a supervised machine learning solution that takes a model architecture and hyperparameters and predicts the training and testing performance. Your solution will be evaluated based on the quality of the performance prediction compared to the ground truth. \n\nWe encourage you to ensure your methods generalize well e.g. using train-test splits or cross-validation. We will use a blind test set strategy to evaluate how well your methods generalize (details in evaluation). You are free to use any model or strategy for this project (whether covered in class or not).\n\n### References\n[1] Neural Architecture Optimization, [https://arxiv.org/abs/1808.07233](https://arxiv.org/abs/1808.07233)\n[2] Progressive Neural Architecture Search, [https://arxiv.org/abs/1712.00559](https://arxiv.org/abs/1712.00559)\n[3] Accelerating Neural Architecture Search using Performance Prediction, [https://arxiv.org/abs/1705.10823](https://arxiv.org/abs/1705.10823)\n[4] A Surprising Linear Relationship Predicts Test Performance in Deep Networks, [https://arxiv.org/abs/1807.09659](https://arxiv.org/abs/1807.09659)\n[5] Predicting the Generalization Gap in Deep Networks with Margin Distributions, [https://arxiv.org/abs/1810.00113](https://arxiv.org/abs/1810.00113)\n\n# Learning Goals\n* Gain real-world experience with ML, subject to blind test evaluation\n* Explore methods for learning with non-vector data e.g. embeddings\n* Gain real-world experience with data preprocessing/cleaning, if necessary\n\n# Kaggle competition and grading\nYour performance will be evaluated via a Kaggle competition. You may sign up for the Kaggle competition here: [https://kaggle.com/c/cs446-fa19](https://kaggle.com/c/cs446-fa19)\n\nWe will identify you using your NetID. An account created using your NetID is preferable for easy matching during the grading time; however, you may use your existing Kaggle account. Please make sure to fill out the signup form below which includes username details.\n\nYou will be restricted to a maximum of 4 submissions per day. The leaderboard shows scores on the validation set. Final grades are based on scores evaluated on a hidden test set with similar distributions as the training and the validation set. \n\nGrading: Your project grade will be determined by thresholds, i.e., you achieve a given grade when your performance is better than the appropriate threshold on the hidden test set. Thresholds will be announced near the end of the project competition. Selected grade thresholds will roughly depend on class performance and our internal baseline tests.\n\n<h1><center>Due date: Competition Ends Dec 12, 11:59 PM</center></h1>\n\n# Data\nThe dataset includes a collection of textual descriptions of neural network model architectures trained on Cifar-10, i.e., inputs “X”. The labels “Y” are the final training and testing performance scores of these models. Do not train your own models on cifar-10, you are provided with training and test examples. Your goal will be to build a performance predictor for this data set.\n\n## Provided Features\nThe performance of a neural network often depends on a few factors [4,5], some of which we have selected as features:\n1.) Architecture description as a string and its hyperparameters\n2.) The first X epochs of training and validation error history\n3.) Initialization statistics. Specifically mean, std and L2 norm of the network for each layer before training starts\n\nWhile your goal is to predict the final train error and final test error, we have interleaved (and identified) examples corresponding to training and testing error. You may consider training a separate performance prediction for the train and test error.\n\n<!-- **Download data here:** [https://uofi.box.com/v/cs446-fa19-project](https://uofi.box.com/v/cs446-fa19-project)\nNOTE: We will also host the data on Azure within a week -->\n\n# Prediction:\nTest labels are submitted as a csv file to the Kaggle site in the format\n**[id, predicted score]**\nNote that the training error and test error scores are interleaved and evaluated together.\n\n<!-- # Evaluation\nThe evaluation metric we will be using is $R^2$ ([Coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination)) which is given by the following equation.\n\n$$ R^2 = 1 - \\frac{SS_{res}}{SS_{tot}} $$\n\nIn this equation $SS_{res} = \\sum_i \\left(y_i - f_i\\right)^2 is the sum of residuals, where $y_i$ is the expected output and $f_i$ is the predicted output. The other term, $SS_{tot} = \\sum_i \\left(y_i - \\bar{y} \\right)^2 is the total sum of squares, where $\\bar{y}$ is the mean of the expected output. -->\n\n# Computational Resources:\nYou are free to use any compute resources you like. Microsoft Azure has graciously donated cloud computing credits for each student to use. If you choose to use these, please see the instructions below.\n\nAnother great source of compute for the project is [**Google Colab**](https://colab.research.google.com/). It is free to use and gives you access to a Telsa K80 GPU. Set up for this is a bit easier than Azure but in order to run your code you must be in a Colab notebook (same as a Jupyter notebook).\n\nAfter filling up the Google form (informing us you will use Azure), please signup for a free trial of azure: [https://azure.microsoft.com/en-us/free/](https://azure.microsoft.com/en-us/free/). We will add you to the course compute group to access additional computing resources.\n\n**Kaggle username declaration and computation signup form [NOTE: FORM CLOSES NOV 28].**\nPlease sign up for computation (if desired) in this form as well: [https://forms.gle/KhkpMAvmJ2c3sVAy7](https://forms.gle/KhkpMAvmJ2c3sVAy7).",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    }
  ]
}