Hi everyone,
\nThis is great. I'm relieved to have maintained second place in the final update, as I was nervous that anything could happen. By my estimate, there was a 60% chance I would drop below 2nd on the 6th update, due to the noisy nature of this kind of data. I also estimate that on a 7th update, Meme Lord Capital would have a 75% chance of maintaining first place. They finished with a big lead on everyone. Good job, Meme Lord Capital!
\nI'd like to thank G-Research for sponsoring the competition, and everyone involved in organizing and hosting it.
\nIn this writeup I'll strive to provide insight into my methods, but without giving away model details that could be profitable for the host. When a good financial indicator becomes common knowledge, everyone uses it, therefore it loses profitability. As such, I won't be sharing my code or any specific insight on features.
\nI decided to enter this competition because it concerns asset price prediction using actual market data (not pre-engineered features). That's the kind of data I like: having an explicit time dimension, many interacting entities, a lot of noise, and a need for creative method development. In 2014-2017, I researched stock price analysis and prediction in my own time. It didn't end up being profitable long-term as a single DIY investor, but I did learn a lot about predicting time series and asset prices.
\nMy toolkit consisted of CPython, Numba, Jupyter, Pandas, LightGBM, Matplotlib, and scikit-learn. Using the Numba compiler instead of CPython for feature generation resulted in high performance. Doing so, I was able to experiment with a wide variety of features, utilizing the entire dataset and iterating quickly.
\nSetting up good N-fold cross validation (CV) is essential. We should try to see if our ideas fail in the most real environment we can make. Every time we have a research question, it should (eventually, after a period of creative exploration) be answered in that manner. If we don't have a good CV setup, our decisions will mostly be mere guesses. Good N-fold CV is easy to set up, taking less than 100 lines of code, without using scikit-learn or any timeseries framework.
\nIn this competition I used 6-fold, walk-forward, grouped cross validation. The group key was the timestamp. In a typical setup, train folds were 40 weeks long, test folds were 40 weeks long, there was a gap of 1 week between test and train folds, and the ends of training folds were incremented by 20 weeks each fold.
\nI chose to overlap my folds so that they could be long, but I could still have 6 folds. With non-overlapping folds, I would have to decide between many short folds or a few long ones.
\nThe advantage of having many folds is that average CV scores will have lower variance. On the other hand, the advantage of having long folds is that the model sees more data, so you get a better picture of how it will perform with the full dataset. Instead of choosing between many short folds or a few long ones, I let them overlap so I could have many long folds. I could have used more than 6 folds, or run CV multiple times, changing seeds. I didn't do these things because CV score variance was decently low, and CV was already almost too slow.
\n(If you don't pay attention to CV variance, you could waste days optimizing your model, only to find that your early decisions were based on noise and incorrect. An easy way to see if your CV scores have too much variance is to look at a plot of CV score vs. a parameter you're tuning. A good plot will usually be smooth, with a knee and a plateau, or maybe a peak or a valley. If the plot looks too noisy to clearly see those things, then try it again, with different seeds, and see if you get different results. If you do, then your CV results are too noisy. You can try using more folds, or running CV several times with different random seeds and averaging.)
\nThe one week gap between train and test data was to prevent CV results from being too rosy. With no gap, a model can cheat at the the beginning of the test period, because the end of the train period is very similar. There was no gap between train and test data in the final submission.
\nAs I was developing my model, naturally I wanted to have an idea of how it would place against competitors. Because public leaderboard scores were unrealistically high due to probing and overfitting, I had to find a way to interpret them. So I looked at masters and grandmasters with \"low\" scores. I figured these competitors were good enough to submit good kernels, but had been careful to keep the public leaderboard period out of their training data. I did not expect true scores to go above 0.1, based on experience and discussions here. Plotting the sorted scores of the \"low\" scoring masters and grandmasters, there was a clear plateau around 0.08, as I recall. That was the score I aimed to beat in my local CV.
\nOf course, I did not use the public leaderboard as a guide in any optimizations. That's always a bad idea (see the paragraph above about CV variance). I only made a total of two submissions in the competition.
\nBecause my labor and computational resources are finite, it’s important to know where to direct them. As I developed my model, feature engineering was guided by feature importance. As I made new features, I focused further effort on developing features sets that already performed well (had high importance), or for which transformations led to easy increases in importance.
\nAs I said in the introduction, I won't be discussing specific feature insight or give my code. I'll only be giving my two cents on methods.
\nThe learner was trivial: a LightGBM GBDT regressor with squared loss. There was no ensembling other than the gradient boosting in GBDT. The only parameters I changed from defaults were the number of estimators, number of leaves, and the learning rate. There was no regularization, augmentation, or feature neutralization. I checked with CV whether these things would help, found that they wouldn't, and decided to go with a simpler submission.
\nLightGBM parameters were not tuned exhaustively. Tuning was taking a long time, and CV results were indicating that regularization would have little effect on model performance. So I decided to not push on that wall and instead placed my focus back on feature engineering. CV is essential in making these resource allocation decisions.
\nBecause this was a code competition with a large dataset, performance was crucial. Our kernels were limited to running in 9 hours or less, using up to 16GB of RAM. As we all can attest, it's easy to bump into those limits.
\nThe hardest limitation I dealt with was the 16GB RAM limit. I wanted to train with the entire dataset, because CV had shown me that scores just kept improving with longer training data. To use the entire dataset, I had to find and fix code that resulted in unnecessary RAM copies of arrays. The Pandas operation df.update() was a frustrating offender. The solution was to select small subsets of data, use df.update() on those subsets, then append to a list of dataframes, and then at the end, concatenate the list into one dataframe.
\nOn the other hand, my kernel is really fast. It takes 25 minutes to import data, generate features, and train the model. When submitting, predictions are generated at a rate of about 50 timestamps per second, not including the overhead of the submission mechanism. Generating predictions for 2 weeks of data takes around 10 minutes, on top of the the 25 minutes to train the model.
\nSpeed wasn't a limiting factor in my submission because I used Numba to write feature generation code and my learner was simple. My main goal was to have a codebase that allowed me to test feature ideas quickly and easily. The fast submission kernel was a side effect.
", "messages": [ { "id": "1777778", "postDate": "05/04/2022 19:21:56", "content": "Hi everyone,
\nThis is great. I'm relieved to have maintained second place in the final update, as I was nervous that anything could happen. By my estimate, there was a 60% chance I would drop below 2nd on the 6th update, due to the noisy nature of this kind of data. I also estimate that on a 7th update, Meme Lord Capital would have a 75% chance of maintaining first place. They finished with a big lead on everyone. Good job, Meme Lord Capital!
\nI'd like to thank G-Research for sponsoring the competition, and everyone involved in organizing and hosting it.
\nIn this writeup I'll strive to provide insight into my methods, but without giving away model details that could be profitable for the host. When a good financial indicator becomes common knowledge, everyone uses it, therefore it loses profitability. As such, I won't be sharing my code or any specific insight on features.
\nI decided to enter this competition because it concerns asset price prediction using actual market data (not pre-engineered features). That's the kind of data I like: having an explicit time dimension, many interacting entities, a lot of noise, and a need for creative method development. In 2014-2017, I researched stock price analysis and prediction in my own time. It didn't end up being profitable long-term as a single DIY investor, but I did learn a lot about predicting time series and asset prices.
\nMy toolkit consisted of CPython, Numba, Jupyter, Pandas, LightGBM, Matplotlib, and scikit-learn. Using the Numba compiler instead of CPython for feature generation resulted in high performance. Doing so, I was able to experiment with a wide variety of features, utilizing the entire dataset and iterating quickly.
\nSetting up good N-fold cross validation (CV) is essential. We should try to see if our ideas fail in the most real environment we can make. Every time we have a research question, it should (eventually, after a period of creative exploration) be answered in that manner. If we don't have a good CV setup, our decisions will mostly be mere guesses. Good N-fold CV is easy to set up, taking less than 100 lines of code, without using scikit-learn or any timeseries framework.
\nIn this competition I used 6-fold, walk-forward, grouped cross validation. The group key was the timestamp. In a typical setup, train folds were 40 weeks long, test folds were 40 weeks long, there was a gap of 1 week between test and train folds, and the ends of training folds were incremented by 20 weeks each fold.
\nI chose to overlap my folds so that they could be long, but I could still have 6 folds. With non-overlapping folds, I would have to decide between many short folds or a few long ones.
\nThe advantage of having many folds is that average CV scores will have lower variance. On the other hand, the advantage of having long folds is that the model sees more data, so you get a better picture of how it will perform with the full dataset. Instead of choosing between many short folds or a few long ones, I let them overlap so I could have many long folds. I could have used more than 6 folds, or run CV multiple times, changing seeds. I didn't do these things because CV score variance was decently low, and CV was already almost too slow.
\n(If you don't pay attention to CV variance, you could waste days optimizing your model, only to find that your early decisions were based on noise and incorrect. An easy way to see if your CV scores have too much variance is to look at a plot of CV score vs. a parameter you're tuning. A good plot will usually be smooth, with a knee and a plateau, or maybe a peak or a valley. If the plot looks too noisy to clearly see those things, then try it again, with different seeds, and see if you get different results. If you do, then your CV results are too noisy. You can try using more folds, or running CV several times with different random seeds and averaging.)
\nThe one week gap between train and test data was to prevent CV results from being too rosy. With no gap, a model can cheat at the the beginning of the test period, because the end of the train period is very similar. There was no gap between train and test data in the final submission.
\nAs I was developing my model, naturally I wanted to have an idea of how it would place against competitors. Because public leaderboard scores were unrealistically high due to probing and overfitting, I had to find a way to interpret them. So I looked at masters and grandmasters with \"low\" scores. I figured these competitors were good enough to submit good kernels, but had been careful to keep the public leaderboard period out of their training data. I did not expect true scores to go above 0.1, based on experience and discussions here. Plotting the sorted scores of the \"low\" scoring masters and grandmasters, there was a clear plateau around 0.08, as I recall. That was the score I aimed to beat in my local CV.
\nOf course, I did not use the public leaderboard as a guide in any optimizations. That's always a bad idea (see the paragraph above about CV variance). I only made a total of two submissions in the competition.
\nBecause my labor and computational resources are finite, it’s important to know where to direct them. As I developed my model, feature engineering was guided by feature importance. As I made new features, I focused further effort on developing features sets that already performed well (had high importance), or for which transformations led to easy increases in importance.
\nAs I said in the introduction, I won't be discussing specific feature insight or give my code. I'll only be giving my two cents on methods.
\nThe learner was trivial: a LightGBM GBDT regressor with squared loss. There was no ensembling other than the gradient boosting in GBDT. The only parameters I changed from defaults were the number of estimators, number of leaves, and the learning rate. There was no regularization, augmentation, or feature neutralization. I checked with CV whether these things would help, found that they wouldn't, and decided to go with a simpler submission.
\nLightGBM parameters were not tuned exhaustively. Tuning was taking a long time, and CV results were indicating that regularization would have little effect on model performance. So I decided to not push on that wall and instead placed my focus back on feature engineering. CV is essential in making these resource allocation decisions.
\nBecause this was a code competition with a large dataset, performance was crucial. Our kernels were limited to running in 9 hours or less, using up to 16GB of RAM. As we all can attest, it's easy to bump into those limits.
\nThe hardest limitation I dealt with was the 16GB RAM limit. I wanted to train with the entire dataset, because CV had shown me that scores just kept improving with longer training data. To use the entire dataset, I had to find and fix code that resulted in unnecessary RAM copies of arrays. The Pandas operation df.update() was a frustrating offender. The solution was to select small subsets of data, use df.update() on those subsets, then append to a list of dataframes, and then at the end, concatenate the list into one dataframe.
\nOn the other hand, my kernel is really fast. It takes 25 minutes to import data, generate features, and train the model. When submitting, predictions are generated at a rate of about 50 timestamps per second, not including the overhead of the submission mechanism. Generating predictions for 2 weeks of data takes around 10 minutes, on top of the the 25 minutes to train the model.
\nSpeed wasn't a limiting factor in my submission because I used Numba to write feature generation code and my learner was simple. My main goal was to have a codebase that allowed me to test feature ideas quickly and easily. The fast submission kernel was a side effect.
", "rawMarkdown": "Hi everyone,\n\nThis is great. I'm relieved to have maintained second place in the final update, as I was nervous that anything could happen. By my estimate, there was a 60% chance I would drop below 2nd on the 6th update, due to the noisy nature of this kind of data. I also estimate that on a 7th update, Meme Lord Capital would have a 75% chance of maintaining first place. They finished with a big lead on everyone. Good job, Meme Lord Capital!\n\nI'd like to thank G-Research for sponsoring the competition, and everyone involved in organizing and hosting it.\n\n# Introduction\n\nIn this writeup I'll strive to provide insight into my methods, but without giving away model details that could be profitable for the host. When a good financial indicator becomes common knowledge, everyone uses it, therefore it loses profitability. As such, I won't be sharing my code or any specific insight on features. \n\nI decided to enter this competition because it concerns asset price prediction using actual market data (not pre-engineered features). That's the kind of data I like: having an explicit time dimension, many interacting entities, a lot of noise, and a need for creative method development. In 2014-2017, I researched stock price analysis and prediction in my own time. It didn't end up being profitable long-term as a single DIY investor, but I did learn a lot about predicting time series and asset prices.\n\n# Toolkit\n\nMy toolkit consisted of CPython, Numba, Jupyter, Pandas, LightGBM, Matplotlib, and scikit-learn. Using the Numba compiler instead of CPython for feature generation resulted in high performance. Doing so, I was able to experiment with a wide variety of features, utilizing the entire dataset and iterating quickly.\n\n# Cross validation\n\nSetting up good N-fold cross validation (CV) is essential. We should try to see if our ideas fail in the most real environment we can make. Every time we have a research question, it should (eventually, after a period of creative exploration) be answered in that manner. If we don't have a good CV setup, our decisions will mostly be mere guesses. Good N-fold CV is easy to set up, taking less than 100 lines of code, without using scikit-learn or any timeseries framework. \n\nIn this competition I used 6-fold, walk-forward, grouped cross validation. The group key was the timestamp. In a typical setup, train folds were 40 weeks long, test folds were 40 weeks long, there was a gap of 1 week between test and train folds, and the ends of training folds were incremented by 20 weeks each fold. \n\nI chose to overlap my folds so that they could be long, but I could still have 6 folds. With non-overlapping folds, I would have to decide between many short folds or a few long ones. \n\nThe advantage of having many folds is that average CV scores will have lower variance. On the other hand, the advantage of having long folds is that the model sees more data, so you get a better picture of how it will perform with the full dataset. Instead of choosing between many short folds or a few long ones, I let them overlap so I could have many long folds. I could have used more than 6 folds, or run CV multiple times, changing seeds. I didn't do these things because CV score variance was decently low, and CV was already almost too slow.\n\n(If you don't pay attention to CV variance, you could waste days optimizing your model, only to find that your early decisions were based on noise and incorrect. An easy way to see if your CV scores have too much variance is to look at a plot of CV score vs. a parameter you're tuning. A good plot will usually be smooth, with a knee and a plateau, or maybe a peak or a valley. If the plot looks too noisy to clearly see those things, then try it again, with different seeds, and see if you get different results. If you do, then your CV results are too noisy. You can try using more folds, or running CV several times with different random seeds and averaging.)\n\nThe one week gap between train and test data was to prevent CV results from being too rosy. With no gap, a model can cheat at the the beginning of the test period, because the end of the train period is very similar. There was no gap between train and test data in the final submission.\n\n# Public leaderboard scores\n\nAs I was developing my model, naturally I wanted to have an idea of how it would place against competitors. Because public leaderboard scores were unrealistically high due to probing and overfitting, I had to find a way to interpret them. So I looked at masters and grandmasters with \"low\" scores. I figured these competitors were good enough to submit good kernels, but had been careful to keep the public leaderboard period out of their training data. I did not expect true scores to go above 0.1, based on experience and discussions here. Plotting the sorted scores of the \"low\" scoring masters and grandmasters, there was a clear plateau around 0.08, as I recall. That was the score I aimed to beat in my local CV. \n\nOf course, I did not use the public leaderboard as a guide in any optimizations. That's always a bad idea (see the paragraph above about CV variance). I only made a total of two submissions in the competition.\n\n# Feature engineering\n\nBecause my labor and computational resources are finite, it’s important to know where to direct them. As I developed my model, feature engineering was guided by feature importance. As I made new features, I focused further effort on developing features sets that already performed well (had high importance), or for which transformations led to easy increases in importance. \n\nAs I said in the introduction, I won't be discussing specific feature insight or give my code. I'll only be giving my two cents on methods.\n\n# Learner\n\nThe learner was trivial: a LightGBM GBDT regressor with squared loss. There was no ensembling other than the gradient boosting in GBDT. The only parameters I changed from defaults were the number of estimators, number of leaves, and the learning rate. There was no regularization, augmentation, or feature neutralization. I checked with CV whether these things would help, found that they wouldn't, and decided to go with a simpler submission.\n\nLightGBM parameters were not tuned exhaustively. Tuning was taking a long time, and CV results were indicating that regularization would have little effect on model performance. So I decided to not push on that wall and instead placed my focus back on feature engineering. CV is essential in making these resource allocation decisions.\n\n# Submission kernel\n\nBecause this was a code competition with a large dataset, performance was crucial. Our kernels were limited to running in 9 hours or less, using up to 16GB of RAM. As we all can attest, it's easy to bump into those limits.\n\nThe hardest limitation I dealt with was the 16GB RAM limit. I wanted to train with the entire dataset, because CV had shown me that scores just kept improving with longer training data. To use the entire dataset, I had to find and fix code that resulted in unnecessary RAM copies of arrays. The Pandas operation df.update() was a frustrating offender. The solution was to select small subsets of data, use df.update() on those subsets, then append to a list of dataframes, and then at the end, concatenate the list into one dataframe.\n\nOn the other hand, my kernel is really fast. It takes 25 minutes to import data, generate features, and train the model. When submitting, predictions are generated at a rate of about 50 timestamps per second, not including the overhead of the submission mechanism. Generating predictions for 2 weeks of data takes around 10 minutes, on top of the the 25 minutes to train the model.\n\nSpeed wasn't a limiting factor in my submission because I used Numba to write feature generation code and my learner was simple. My main goal was to have a codebase that allowed me to test feature ideas quickly and easily. The fast submission kernel was a side effect.", "votes": null }, { "id": "1777798", "postDate": "05/04/2022 19:38:52", "content": "Anything you would have like to explored if you had more resources and time?
", "rawMarkdown": "Anything you would have like to explored if you had more resources and time?", "votes": null }, { "id": "1777817", "postDate": "05/04/2022 20:03:37", "content": "@nathanrm Wow congratulations on getting 2nd place, amazing work
", "rawMarkdown": "nathanrm Wow congratulations on getting 2nd place, amazing work", "votes": null }, { "id": "1777831", "postDate": "05/04/2022 20:11:58", "content": "Congratulations! I know you said you don't want to go into your FE or other details, but I just wonder, how much do you think your success can be attributed to model choice/hyperparameter choice/cross validation setup, and how much is feature engineering? Do you think you that finding the right features was the key in this competition? As you said you did not any \"fancy\" models, I assume there must be something special in your feature engineering.
", "rawMarkdown": "Congratulations! I know you said you don't want to go into your FE or other details, but I just wonder, how much do you think your success can be attributed to model choice/hyperparameter choice/cross validation setup, and how much is feature engineering? Do you think you that finding the right features was the key in this competition? As you said you did not any \"fancy\" models, I assume there must be something special in your feature engineering.", "votes": null }, { "id": "1777880", "postDate": "05/04/2022 21:42:25", "content": "Absolutely, there are a lot of things.
\nAll the way up to the deadline, feature engineering was the best avenue for improvement. So I'd continue developing features. I'd want to improve my codebase a little first.
I'd revisit some decisions I made based on CV. With more computational resources, CV scores would have less variance, and I might be able to see, for example, that regularization actually improved the model a little.
I'd try more complex models, especially using autoencoders, convolutional NN's, recursive NN's, etc.
There's a feature engineering research idea I'd like to try, but I probably shouldn't discuss it publicly.
This competition got me interested again in something I was researching a while ago. Essentially, I\"m thinking of supervised learning problems where the data is nonlinear, high dimensional and has a lot of noise. There are toy datasets that seem like they should be easy to fit, but quickly become challenging as noisy dimensions are added. Neural nets do the best on these toy datasets, with decision forest methods coming in a distant second, and basically every other method fails with even a few dimensions. It's interesting to try to make new models that do better than neural nets in that playground.
I particularly enjoy a certain area of research that has applications to asset price prediction. That's unsupervised data decompositions in general (PCA being an example), but especially if there's any connection to statistical mechanics, quantum mechancs, information theory, or emergence.
It was mostly feature engineering, and yes, finding the right features was key.
", "rawMarkdown": "It was mostly feature engineering, and yes, finding the right features was key.", "votes": null }, { "id": "1777891", "postDate": "05/04/2022 21:56:13", "content": "Congratulations and thanks for sharing!
\n\n\nThere was no ensembling
\n
Impressive!
", "rawMarkdown": "Congratulations and thanks for sharing!\n\n>There was no ensembling\n\nImpressive!", "votes": null }, { "id": "1778006", "postDate": "05/05/2022 02:45:35", "content": "Congratulations!
\nI believe feature engineering is the key. :)
@nathanrm congratulations! very impressive :)
", "rawMarkdown": "nathanrm congratulations! very impressive :)", "votes": null }, { "id": "1778393", "postDate": "05/05/2022 09:22:58", "content": "Congratulation. Nice to see Feature Engineering and (mostly untuned) lgbm for the win.
\nOf course I have some questions regarding modelling, but feel free to not answer; How do you handle coind id ? (one model for all coin ? with the coin as a feature ? or individual models ?). Regarding FE, what would be your opinion on what has been shared ? (are common factor a good idea ? technical analysis ? or did you go into more complex stuff ?)
", "rawMarkdown": "Congratulation. Nice to see Feature Engineering and (mostly untuned) lgbm for the win. \n\nOf course I have some questions regarding modelling, but feel free to not answer; How do you handle coind id ? (one model for all coin ? with the coin as a feature ? or individual models ?). Regarding FE, what would be your opinion on what has been shared ? (are common factor a good idea ? technical analysis ? or did you go into more complex stuff ?)", "votes": null }, { "id": "1778615", "postDate": "05/05/2022 14:06:59", "content": "This is a great post! Thank you for sharing so much information. Your writeup has given me a lot to think about in terms of how I approach future data competitions.
", "rawMarkdown": "This is a great post! Thank you for sharing so much information. Your writeup has given me a lot to think about in terms of how I approach future data competitions.", "votes": null }, { "id": "1778652", "postDate": "05/05/2022 14:57:16", "content": "Very good job and congratz!
\nCould you share any further insight into how you went on to select meaningful features? Was it just adding and subtracting features, looking at lgbm feature importance etc. Or something more sophisticated?
", "rawMarkdown": "Very good job and congratz! \n\nCould you share any further insight into how you went on to select meaningful features? Was it just adding and subtracting features, looking at lgbm feature importance etc. Or something more sophisticated?", "votes": null }, { "id": "1779424", "postDate": "05/06/2022 12:25:04", "content": "Congratulations @nathanrm and thanks for sharing the solution summary.
", "rawMarkdown": "Congratulations @nathanrm and thanks for sharing the solution summary.", "votes": null }, { "id": "1779456", "postDate": "05/06/2022 13:15:02", "content": "Congrats! I would have loved to hear a bit more about the feature engineering part but also understand why you are keeping that secret. Well done!
", "rawMarkdown": "Congrats! I would have loved to hear a bit more about the feature engineering part but also understand why you are keeping that secret. Well done!", "votes": null }, { "id": "1779601", "postDate": "05/06/2022 16:03:57", "content": "That's a good question. You guessed a topic that I decided to leave out. It was more sophisticated.
", "rawMarkdown": "That's a good question. You guessed a topic that I decided to leave out. It was more sophisticated.", "votes": null }, { "id": "1779602", "postDate": "05/06/2022 16:06:55", "content": "Good questions. You too guessed a topic I decided to leave out. I tried all of it. CV and feature importance guided the way.
", "rawMarkdown": "Good questions. You too guessed a topic I decided to leave out. I tried all of it. CV and feature importance guided the way.", "votes": null }, { "id": "1779873", "postDate": "05/06/2022 22:52:57", "content": "congrats mate
", "rawMarkdown": "congrats mate", "votes": null }, { "id": "1782007", "postDate": "05/09/2022 06:43:26", "content": "Simple but useful suggestion,thanks sharing!
", "rawMarkdown": "Simple but useful suggestion,thanks sharing!", "votes": null }, { "id": "1782754", "postDate": "05/09/2022 20:41:15", "content": "Any recommendation/resources for learning about time series and asset prediction for a newbie?
", "rawMarkdown": "Any recommendation/resources for learning about time series and asset prediction for a newbie?", "votes": null }, { "id": "1782908", "postDate": "05/10/2022 01:50:31", "content": "Congraturation!!
", "rawMarkdown": "Congraturation!!", "votes": null }, { "id": "1796589", "postDate": "05/21/2022 01:08:52", "content": "Giving a recommendation without knowing your background would be difficult. You could search \"best timeseries book\" and find one that's highly recommended, specific to your programming language and application domain. I come from a physics background, and timeseries are a natural extension of that, so I learned by experimenting and reading here and there.
\nTimeseries learning is essentially supervised learning that respects causality. You avoid doing anything in your algorithm that would depend on data from the future. That's both harder and easier than it sounds. Technically, it's easy because you're avoiding doing something. In practice, you can sometimes take shortcuts and make mistakes that end up causing your algorithm to need future data. Learning how to avoid these pitfalls is one of the main things to learning timeseries forecasting. You want to be as creative as possible, clustering assets by market, etc., without making these mistakes and breaking causality. You have to develop a way of thinking about it. The only way I know to do that is with experience, but there might be a resource that teaches that way of thinking.
\nThere are some things to learn that are very useful for timeseries, like de-trending data by differencing, lagged features, etc. A good timeseries book should show you these things, but a lot of it is almost self-suggesting, based on understanding the data you're modelling. For example, you could learn on your own about de-trending data by diagnosing a timeseries algorithm that didn't work (and then you'd also learn about diagnosing timeseries models).
\nBecause understanding the data is important, domain knowledge is important.
\nThe rest of it (actually a huge portion of it) is just machine learning. Every technique, method, or algorithm that can be applied to a supervised or unsupervised learning problem can be used for timeseries prediction.
", "rawMarkdown": "Giving a recommendation without knowing your background would be difficult. You could search \"best timeseries book\" and find one that's highly recommended, specific to your programming language and application domain. I come from a physics background, and timeseries are a natural extension of that, so I learned by experimenting and reading here and there. \n\nTimeseries learning is essentially supervised learning that respects causality. You avoid doing anything in your algorithm that would depend on data from the future. That's both harder and easier than it sounds. Technically, it's easy because you're avoiding doing something. In practice, you can sometimes take shortcuts and make mistakes that end up causing your algorithm to need future data. Learning how to avoid these pitfalls is one of the main things to learning timeseries forecasting. You want to be as creative as possible, clustering assets by market, etc., without making these mistakes and breaking causality. You have to develop a way of thinking about it. The only way I know to do that is with experience, but there might be a resource that teaches that way of thinking.\n\nThere are some things to learn that are very useful for timeseries, like de-trending data by differencing, lagged features, etc. A good timeseries book should show you these things, but a lot of it is almost self-suggesting, based on understanding the data you're modelling. For example, you could learn on your own about de-trending data by diagnosing a timeseries algorithm that didn't work (and then you'd also learn about diagnosing timeseries models). \n\nBecause understanding the data is important, domain knowledge is important.\n\nThe rest of it (actually a huge portion of it) is just machine learning. Every technique, method, or algorithm that can be applied to a supervised or unsupervised learning problem can be used for timeseries prediction.", "votes": null }, { "id": "2159495", "postDate": "02/25/2023 19:31:27", "content": "Could you tell us, nearly 1 year after the competition, if your model was able to significantly outperform the market?
", "rawMarkdown": "Could you tell us, nearly 1 year after the competition, if your model was able to significantly outperform the market?", "votes": null }, { "id": "2253109", "postDate": "05/09/2023 23:26:02", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": null }, { "id": "2280064", "postDate": "05/29/2023 20:44:15", "content": "This is something I also want to know
", "rawMarkdown": "This is something I also want to know", "votes": null }, { "id": "3211984", "postDate": "05/29/2025 06:15:26", "content": "Will your model is able to predict short term crypto price movement ?
", "rawMarkdown": "Will your model is able to predict short term crypto price movement ?", "votes": null }, { "id": "3525900", "postDate": "09/19/2026 12:28:06", "content": "hello any update so far on the models oos performance
", "rawMarkdown": "hello any update so far on the models oos performance", "votes": null } ], "comments": [ { "id": 1777798, "author_name": "valindor", "author_url": "", "post_date": "05/04/2022 19:38:52", "content": "Anything you would have like to explored if you had more resources and time?
", "votes": null, "replies": [ { "id": 1777880, "author_name": "nathanrm", "author_url": "", "post_date": "05/04/2022 21:42:25", "content": "Absolutely, there are a lot of things.
\nAll the way up to the deadline, feature engineering was the best avenue for improvement. So I'd continue developing features. I'd want to improve my codebase a little first.
I'd revisit some decisions I made based on CV. With more computational resources, CV scores would have less variance, and I might be able to see, for example, that regularization actually improved the model a little.
I'd try more complex models, especially using autoencoders, convolutional NN's, recursive NN's, etc.
There's a feature engineering research idea I'd like to try, but I probably shouldn't discuss it publicly.
This competition got me interested again in something I was researching a while ago. Essentially, I\"m thinking of supervised learning problems where the data is nonlinear, high dimensional and has a lot of noise. There are toy datasets that seem like they should be easy to fit, but quickly become challenging as noisy dimensions are added. Neural nets do the best on these toy datasets, with decision forest methods coming in a distant second, and basically every other method fails with even a few dimensions. It's interesting to try to make new models that do better than neural nets in that playground.
I particularly enjoy a certain area of research that has applications to asset price prediction. That's unsupervised data decompositions in general (PCA being an example), but especially if there's any connection to statistical mechanics, quantum mechancs, information theory, or emergence.
@nathanrm Wow congratulations on getting 2nd place, amazing work
", "votes": null, "replies": [] }, { "id": 1777831, "author_name": "fritzcremer", "author_url": "", "post_date": "05/04/2022 20:11:58", "content": "Congratulations! I know you said you don't want to go into your FE or other details, but I just wonder, how much do you think your success can be attributed to model choice/hyperparameter choice/cross validation setup, and how much is feature engineering? Do you think you that finding the right features was the key in this competition? As you said you did not any \"fancy\" models, I assume there must be something special in your feature engineering.
", "votes": null, "replies": [ { "id": 1777881, "author_name": "nathanrm", "author_url": "", "post_date": "05/04/2022 21:43:09", "content": "It was mostly feature engineering, and yes, finding the right features was key.
", "votes": null, "replies": [] } ] }, { "id": 1777891, "author_name": "julianmukaj", "author_url": "", "post_date": "05/04/2022 21:56:13", "content": "Congratulations and thanks for sharing!
\n\n\nThere was no ensembling
\n
Impressive!
", "votes": null, "replies": [] }, { "id": 1778006, "author_name": "kakarroto", "author_url": "", "post_date": "05/05/2022 02:45:35", "content": "Congratulations!
\nI believe feature engineering is the key. :)
@nathanrm congratulations! very impressive :)
", "votes": null, "replies": [] }, { "id": 1778393, "author_name": "lucasmorin", "author_url": "", "post_date": "05/05/2022 09:22:58", "content": "Congratulation. Nice to see Feature Engineering and (mostly untuned) lgbm for the win.
\nOf course I have some questions regarding modelling, but feel free to not answer; How do you handle coind id ? (one model for all coin ? with the coin as a feature ? or individual models ?). Regarding FE, what would be your opinion on what has been shared ? (are common factor a good idea ? technical analysis ? or did you go into more complex stuff ?)
", "votes": null, "replies": [ { "id": 1779602, "author_name": "nathanrm", "author_url": "", "post_date": "05/06/2022 16:06:55", "content": "Good questions. You too guessed a topic I decided to leave out. I tried all of it. CV and feature importance guided the way.
", "votes": null, "replies": [] } ] }, { "id": 1778615, "author_name": "", "author_url": "", "post_date": "05/05/2022 14:06:59", "content": "This is a great post! Thank you for sharing so much information. Your writeup has given me a lot to think about in terms of how I approach future data competitions.
", "votes": null, "replies": [] }, { "id": 1778652, "author_name": "vuk1998", "author_url": "", "post_date": "05/05/2022 14:57:16", "content": "Very good job and congratz!
\nCould you share any further insight into how you went on to select meaningful features? Was it just adding and subtracting features, looking at lgbm feature importance etc. Or something more sophisticated?
", "votes": null, "replies": [ { "id": 1779601, "author_name": "nathanrm", "author_url": "", "post_date": "05/06/2022 16:03:57", "content": "That's a good question. You guessed a topic that I decided to leave out. It was more sophisticated.
", "votes": null, "replies": [] } ] }, { "id": 1779424, "author_name": "sudalairajkumar", "author_url": "", "post_date": "05/06/2022 12:25:04", "content": "Congratulations @nathanrm and thanks for sharing the solution summary.
", "votes": null, "replies": [] }, { "id": 1779456, "author_name": "loicge", "author_url": "", "post_date": "05/06/2022 13:15:02", "content": "Congrats! I would have loved to hear a bit more about the feature engineering part but also understand why you are keeping that secret. Well done!
", "votes": null, "replies": [] }, { "id": 1779873, "author_name": "mariaaboulaala", "author_url": "", "post_date": "05/06/2022 22:52:57", "content": "congrats mate
", "votes": null, "replies": [] }, { "id": 1782007, "author_name": "arthurhsu", "author_url": "", "post_date": "05/09/2022 06:43:26", "content": "Simple but useful suggestion,thanks sharing!
", "votes": null, "replies": [] }, { "id": 1782754, "author_name": "psiquer", "author_url": "", "post_date": "05/09/2022 20:41:15", "content": "Any recommendation/resources for learning about time series and asset prediction for a newbie?
", "votes": null, "replies": [ { "id": 1796589, "author_name": "nathanrm", "author_url": "", "post_date": "05/21/2022 01:08:52", "content": "Giving a recommendation without knowing your background would be difficult. You could search \"best timeseries book\" and find one that's highly recommended, specific to your programming language and application domain. I come from a physics background, and timeseries are a natural extension of that, so I learned by experimenting and reading here and there.
\nTimeseries learning is essentially supervised learning that respects causality. You avoid doing anything in your algorithm that would depend on data from the future. That's both harder and easier than it sounds. Technically, it's easy because you're avoiding doing something. In practice, you can sometimes take shortcuts and make mistakes that end up causing your algorithm to need future data. Learning how to avoid these pitfalls is one of the main things to learning timeseries forecasting. You want to be as creative as possible, clustering assets by market, etc., without making these mistakes and breaking causality. You have to develop a way of thinking about it. The only way I know to do that is with experience, but there might be a resource that teaches that way of thinking.
\nThere are some things to learn that are very useful for timeseries, like de-trending data by differencing, lagged features, etc. A good timeseries book should show you these things, but a lot of it is almost self-suggesting, based on understanding the data you're modelling. For example, you could learn on your own about de-trending data by diagnosing a timeseries algorithm that didn't work (and then you'd also learn about diagnosing timeseries models).
\nBecause understanding the data is important, domain knowledge is important.
\nThe rest of it (actually a huge portion of it) is just machine learning. Every technique, method, or algorithm that can be applied to a supervised or unsupervised learning problem can be used for timeseries prediction.
", "votes": null, "replies": [] } ] }, { "id": 1782908, "author_name": "rlaalsrb3246", "author_url": "", "post_date": "05/10/2022 01:50:31", "content": "Congraturation!!
", "votes": null, "replies": [] }, { "id": 2159495, "author_name": "cryptocoin2132", "author_url": "", "post_date": "02/25/2023 19:31:27", "content": "Could you tell us, nearly 1 year after the competition, if your model was able to significantly outperform the market?
", "votes": null, "replies": [ { "id": 2280064, "author_name": "jenmar", "author_url": "", "post_date": "05/29/2023 20:44:15", "content": "This is something I also want to know
", "votes": null, "replies": [] } ] }, { "id": 2253109, "author_name": "sahandnazarzadeh", "author_url": "", "post_date": "05/09/2023 23:26:02", "content": "Congratulations!
", "votes": null, "replies": [] }, { "id": 3211984, "author_name": "battleninza", "author_url": "", "post_date": "05/29/2025 06:15:26", "content": "Will your model is able to predict short term crypto price movement ?
", "votes": null, "replies": [] }, { "id": 3525900, "author_name": "rooneysamwel", "author_url": "", "post_date": "09/19/2026 12:28:06", "content": "hello any update so far on the models oos performance
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1777778": "Hi everyone,\n\nThis is great. I'm relieved to have maintained second place in the final update, as I was nervous that anything could happen. By my estimate, there was a 60% chance I would drop below 2nd on the 6th update, due to the noisy nature of this kind of data. I also estimate that on a 7th update, Meme Lord Capital would have a 75% chance of maintaining first place. They finished with a big lead on everyone. Good job, Meme Lord Capital!\n\nI'd like to thank G-Research for sponsoring the competition, and everyone involved in organizing and hosting it.\n\n# Introduction\n\nIn this writeup I'll strive to provide insight into my methods, but without giving away model details that could be profitable for the host. When a good financial indicator becomes common knowledge, everyone uses it, therefore it loses profitability. As such, I won't be sharing my code or any specific insight on features. \n\nI decided to enter this competition because it concerns asset price prediction using actual market data (not pre-engineered features). That's the kind of data I like: having an explicit time dimension, many interacting entities, a lot of noise, and a need for creative method development. In 2014-2017, I researched stock price analysis and prediction in my own time. It didn't end up being profitable long-term as a single DIY investor, but I did learn a lot about predicting time series and asset prices.\n\n# Toolkit\n\nMy toolkit consisted of CPython, Numba, Jupyter, Pandas, LightGBM, Matplotlib, and scikit-learn. Using the Numba compiler instead of CPython for feature generation resulted in high performance. Doing so, I was able to experiment with a wide variety of features, utilizing the entire dataset and iterating quickly.\n\n# Cross validation\n\nSetting up good N-fold cross validation (CV) is essential. We should try to see if our ideas fail in the most real environment we can make. Every time we have a research question, it should (eventually, after a period of creative exploration) be answered in that manner. If we don't have a good CV setup, our decisions will mostly be mere guesses. Good N-fold CV is easy to set up, taking less than 100 lines of code, without using scikit-learn or any timeseries framework. \n\nIn this competition I used 6-fold, walk-forward, grouped cross validation. The group key was the timestamp. In a typical setup, train folds were 40 weeks long, test folds were 40 weeks long, there was a gap of 1 week between test and train folds, and the ends of training folds were incremented by 20 weeks each fold. \n\nI chose to overlap my folds so that they could be long, but I could still have 6 folds. With non-overlapping folds, I would have to decide between many short folds or a few long ones. \n\nThe advantage of having many folds is that average CV scores will have lower variance. On the other hand, the advantage of having long folds is that the model sees more data, so you get a better picture of how it will perform with the full dataset. Instead of choosing between many short folds or a few long ones, I let them overlap so I could have many long folds. I could have used more than 6 folds, or run CV multiple times, changing seeds. I didn't do these things because CV score variance was decently low, and CV was already almost too slow.\n\n(If you don't pay attention to CV variance, you could waste days optimizing your model, only to find that your early decisions were based on noise and incorrect. An easy way to see if your CV scores have too much variance is to look at a plot of CV score vs. a parameter you're tuning. A good plot will usually be smooth, with a knee and a plateau, or maybe a peak or a valley. If the plot looks too noisy to clearly see those things, then try it again, with different seeds, and see if you get different results. If you do, then your CV results are too noisy. You can try using more folds, or running CV several times with different random seeds and averaging.)\n\nThe one week gap between train and test data was to prevent CV results from being too rosy. With no gap, a model can cheat at the the beginning of the test period, because the end of the train period is very similar. There was no gap between train and test data in the final submission.\n\n# Public leaderboard scores\n\nAs I was developing my model, naturally I wanted to have an idea of how it would place against competitors. Because public leaderboard scores were unrealistically high due to probing and overfitting, I had to find a way to interpret them. So I looked at masters and grandmasters with \"low\" scores. I figured these competitors were good enough to submit good kernels, but had been careful to keep the public leaderboard period out of their training data. I did not expect true scores to go above 0.1, based on experience and discussions here. Plotting the sorted scores of the \"low\" scoring masters and grandmasters, there was a clear plateau around 0.08, as I recall. That was the score I aimed to beat in my local CV. \n\nOf course, I did not use the public leaderboard as a guide in any optimizations. That's always a bad idea (see the paragraph above about CV variance). I only made a total of two submissions in the competition.\n\n# Feature engineering\n\nBecause my labor and computational resources are finite, it’s important to know where to direct them. As I developed my model, feature engineering was guided by feature importance. As I made new features, I focused further effort on developing features sets that already performed well (had high importance), or for which transformations led to easy increases in importance. \n\nAs I said in the introduction, I won't be discussing specific feature insight or give my code. I'll only be giving my two cents on methods.\n\n# Learner\n\nThe learner was trivial: a LightGBM GBDT regressor with squared loss. There was no ensembling other than the gradient boosting in GBDT. The only parameters I changed from defaults were the number of estimators, number of leaves, and the learning rate. There was no regularization, augmentation, or feature neutralization. I checked with CV whether these things would help, found that they wouldn't, and decided to go with a simpler submission.\n\nLightGBM parameters were not tuned exhaustively. Tuning was taking a long time, and CV results were indicating that regularization would have little effect on model performance. So I decided to not push on that wall and instead placed my focus back on feature engineering. CV is essential in making these resource allocation decisions.\n\n# Submission kernel\n\nBecause this was a code competition with a large dataset, performance was crucial. Our kernels were limited to running in 9 hours or less, using up to 16GB of RAM. As we all can attest, it's easy to bump into those limits.\n\nThe hardest limitation I dealt with was the 16GB RAM limit. I wanted to train with the entire dataset, because CV had shown me that scores just kept improving with longer training data. To use the entire dataset, I had to find and fix code that resulted in unnecessary RAM copies of arrays. The Pandas operation df.update() was a frustrating offender. The solution was to select small subsets of data, use df.update() on those subsets, then append to a list of dataframes, and then at the end, concatenate the list into one dataframe.\n\nOn the other hand, my kernel is really fast. It takes 25 minutes to import data, generate features, and train the model. When submitting, predictions are generated at a rate of about 50 timestamps per second, not including the overhead of the submission mechanism. Generating predictions for 2 weeks of data takes around 10 minutes, on top of the the 25 minutes to train the model.\n\nSpeed wasn't a limiting factor in my submission because I used Numba to write feature generation code and my learner was simple. My main goal was to have a codebase that allowed me to test feature ideas quickly and easily. The fast submission kernel was a side effect.", "1777798": "Anything you would have like to explored if you had more resources and time?", "1777817": "nathanrm Wow congratulations on getting 2nd place, amazing work", "1777831": "Congratulations! I know you said you don't want to go into your FE or other details, but I just wonder, how much do you think your success can be attributed to model choice/hyperparameter choice/cross validation setup, and how much is feature engineering? Do you think you that finding the right features was the key in this competition? As you said you did not any \"fancy\" models, I assume there must be something special in your feature engineering.", "1777880": "Absolutely, there are a lot of things.\n\n1. All the way up to the deadline, feature engineering was the best avenue for improvement. So I'd continue developing features. I'd want to improve my codebase a little first.\n\n2. I'd revisit some decisions I made based on CV. With more computational resources, CV scores would have less variance, and I might be able to see, for example, that regularization actually improved the model a little.\n\n3. I'd try more complex models, especially using autoencoders, convolutional NN's, recursive NN's, etc.\n\n4. There's a feature engineering research idea I'd like to try, but I probably shouldn't discuss it publicly.\n\n5. This competition got me interested again in something I was researching a while ago. Essentially, I\"m thinking of supervised learning problems where the data is nonlinear, high dimensional and has a lot of noise. There are toy datasets that seem like they should be easy to fit, but quickly become challenging as noisy dimensions are added. Neural nets do the best on these toy datasets, with decision forest methods coming in a distant second, and basically every other method fails with even a few dimensions. It's interesting to try to make new models that do better than neural nets in that playground.\n\n6. I particularly enjoy a certain area of research that has applications to asset price prediction. That's unsupervised data decompositions in general (PCA being an example), but especially if there's any connection to statistical mechanics, quantum mechancs, information theory, or emergence.", "1777881": "It was mostly feature engineering, and yes, finding the right features was key.", "1777891": "Congratulations and thanks for sharing!\n\n>There was no ensembling\n\nImpressive!", "1778006": "Congratulations! \nI believe feature engineering is the key. :)", "1778008": "nathanrm congratulations! very impressive :)", "1778393": "Congratulation. Nice to see Feature Engineering and (mostly untuned) lgbm for the win. \n\nOf course I have some questions regarding modelling, but feel free to not answer; How do you handle coind id ? (one model for all coin ? with the coin as a feature ? or individual models ?). Regarding FE, what would be your opinion on what has been shared ? (are common factor a good idea ? technical analysis ? or did you go into more complex stuff ?)", "1778615": "This is a great post! Thank you for sharing so much information. Your writeup has given me a lot to think about in terms of how I approach future data competitions.", "1778652": "Very good job and congratz! \n\nCould you share any further insight into how you went on to select meaningful features? Was it just adding and subtracting features, looking at lgbm feature importance etc. Or something more sophisticated?", "1779424": "Congratulations @nathanrm and thanks for sharing the solution summary.", "1779456": "Congrats! I would have loved to hear a bit more about the feature engineering part but also understand why you are keeping that secret. Well done!", "1779601": "That's a good question. You guessed a topic that I decided to leave out. It was more sophisticated.", "1779602": "Good questions. You too guessed a topic I decided to leave out. I tried all of it. CV and feature importance guided the way.", "1779873": "congrats mate", "1782007": "Simple but useful suggestion,thanks sharing!", "1782754": "Any recommendation/resources for learning about time series and asset prediction for a newbie?", "1782908": "Congraturation!!", "1796589": "Giving a recommendation without knowing your background would be difficult. You could search \"best timeseries book\" and find one that's highly recommended, specific to your programming language and application domain. I come from a physics background, and timeseries are a natural extension of that, so I learned by experimenting and reading here and there. \n\nTimeseries learning is essentially supervised learning that respects causality. You avoid doing anything in your algorithm that would depend on data from the future. That's both harder and easier than it sounds. Technically, it's easy because you're avoiding doing something. In practice, you can sometimes take shortcuts and make mistakes that end up causing your algorithm to need future data. Learning how to avoid these pitfalls is one of the main things to learning timeseries forecasting. You want to be as creative as possible, clustering assets by market, etc., without making these mistakes and breaking causality. You have to develop a way of thinking about it. The only way I know to do that is with experience, but there might be a resource that teaches that way of thinking.\n\nThere are some things to learn that are very useful for timeseries, like de-trending data by differencing, lagged features, etc. A good timeseries book should show you these things, but a lot of it is almost self-suggesting, based on understanding the data you're modelling. For example, you could learn on your own about de-trending data by diagnosing a timeseries algorithm that didn't work (and then you'd also learn about diagnosing timeseries models). \n\nBecause understanding the data is important, domain knowledge is important.\n\nThe rest of it (actually a huge portion of it) is just machine learning. Every technique, method, or algorithm that can be applied to a supervised or unsupervised learning problem can be used for timeseries prediction.", "2159495": "Could you tell us, nearly 1 year after the competition, if your model was able to significantly outperform the market?", "2253109": "Congratulations!", "2280064": "This is something I also want to know", "3211984": "Will your model is able to predict short term crypto price movement ?", "3525900": "hello any update so far on the models oos performance" }, "source": "meta" }