For this Kaggle competition, you will be using 500ms aggregate \"high\" frequency order book data of a futures contract to predict future 1-second price changes.
\n\nEach row of data represents one snapshot in time. For the label \"y\", an entry of \"1\" means the \"mid\" price will go up (strictly up), and \"0\" means the mid price will either not move or go down. You should predict the relative probability of a \"1\" at each timestep of the test set.
\n\nThere can be 12 submissions per day per team and submission window will close on Thursday 13 FEB 2020 at 2:00PM PST.
\n\nYou may not use any additional data sources.
\n\nWe thank Jim for providing this dataset.
\n\n### Non-enrolled entries will be aggregated into a TA benchmark (grey) and will not count towards student scores.\n\nThe average student is not expected to reach this aggregate, since non-enrolled people usually have significantly more experience.", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Evaluation", "content": "The evaluation metric for this competition is the ROC-AUC Score/Metric. This means you can submit probabilities instead of 1's and 0's. More specifically, predict the probability of a \"1\" (mid price strictly increases) occurring.\n\n### The AUC metric\n\nDue to AUC's computation method (vary a decision threshold and calculate an integral), the exact class probability is not important for prediction, only the relative probabilities between the predictions. Unless you are trying to aggregate multiple different model classes, you do not need to worry much about calibrating the exact probability.\n\n
For every timestep (id) in the test.csv dataset, submission files should contain an associated probability of the mid price strictly increasing in the next 2 timesteps. See the sample submission for an example.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Prizes", "content": "#Eternal glory\n\nThe winner(s) will be announced in class on Thursday. This competition and results page will persist for as long as Kaggle (now acquired by Alphabet) does, which means...\n\n#Evidence-backed resume padding\n\nhttps://www.kaggle.com/c/caltech-cs-155-2019-part-1/leaderboard\nhttps://www.kaggle.com/c/caltech-cs-155-2019-final/leaderboard\nhttps://www.kaggle.com/c/caltech-cs-155-2018/leaderboard/\n\nIn the TA's experience, every firm asked about the CS155 Kaggle links and methodology on their resume. If you are interested in similar problems to this Kaggle, contact Jim after the competition.", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Tips", "content": "#Anaconda\n\nIf using Python, pandas is recommended for data manipulation. For prediction, one can start with some typical models in scikit-learn. **pandas, numpy, sklearn are all included in Anaconda.**\n\n#Useful pandas functions\n```\ndf_test = pd.read_csv('filename.csv', index_col=0)\ndf_test['Predicted'] = model.predict_proba(x_test)[:,1]\ndf_test[['Predicted']].to_csv('submission.csv')\n```\n\n#Don't be afraid to try things you haven't heard of before\n\nAfter some basic models, one can progress to some more advanced models in other libraries. Maybe search around a bit for cool or successful models that people have used for classification.\n\n#IDE/Shortcuts\n\nLearn all the shortcuts to your development environment. Spyder and Jupyter are both included in Anaconda, and there are plenty of shortcuts to run specific code sections swiftly.\n\n#Missing data\n\nThink of how to fill in the missing data. Note that the id's for testing data is scrambled, so using the \"previous\" row is not necessarily reasonable.\n\n#Feature engineering\n\nDon't try any timeseries models, but you can certainly engineer a few more features out of the available columns.\n\n#Model blending\n\nWhy use one model when you can take the average of multiple model outputs, or even use another model on top of the model outputs?\n\n#Probability calibration\n\nSome models don't have accurate classification probabilities, but one can calibrate them using various techniques. This doesn't help much for individual models due to the AUC metric, but allows for \"fairer\" model blending if you plan to take a simple average.\n\n#Hyperparameter tuning\n\nTry not to do tuning by hand unless you want to sit at the computer forever. Gain some intuition and then try some automatic tuners! You will likely learn how to save progress to disk.", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false } ] }