Decision Tree Bagging

A few months ago I started looking for a new trading system idea following the same machine learning philosophy as the last one. 

The previous system was based on support vector regression, and used sliding window crossvalidation to set the kernel width and SVM cost parameters. It had a few problems:

1. MLE (maximum likelihood estimation) models are always uncertain because they may not be robust. Look up “model averaging” or “Bayesian methods”

2. Crossvalidation required retraining the SVM thousands of times which was extremely slow.

3. Support Vector Regression can only take numeric/ordered data as inputs, not categorical.

4. Regression estimates can be hard to interpret. e.g. if during training the system never saw a day where the price rose 20%, then it predicts 20% should you interpret it as a strong or a broken signal?

5. Crossvalidation over two parameters required….

A few months ago I started looking for a new trading system idea following the same machine learning philosophy as the last one. 

The previous system was based on support vector regression, and used sliding window crossvalidation to set the kernel width and SVM cost parameters. It had a few problems:

1. MLE (maximum likelihood estimation) models are always uncertain because they may not be robust. Look up “model averaging” or “Bayesian methods”

2. Crossvalidation required retraining the SVM thousands of times which was extremely slow.

3. Support Vector Regression can only take numeric/ordered data as inputs, not categorical.

4. Regression estimates can be hard to interpret. e.g. if during training the system never saw a day where the price rose 20%, then it predicts 20% should you interpret it as a strong or a broken signal?

5. Crossvalidation over two parameters required the creation of a 4 dimensional matrix to store performance. This was very hard to visualize, especially after not working on the system for a while. 

6. Coming up with a loss function for regression is hard for the application of trading. MSE is not perfect, nor is correlation. The loss function should penalize false positives because of transaction costs.

7. No natural confidence values.

8. Hard to interpret SVM. Infinite features w/ Gaussian RBF? The effect of changing C & kernel width are not easy to anticipate or interpret.

9. Non-linear but biased toward linearity (for ex., bad at learning XOR)

A few months ago I settled on a new learning algorithm to build a system on: the random forest. Random forest is a clever name for decision tree bagging (ensemble). And bagging is a clever conjunction of “bootstrap aggregating”. I especially liked that it could accept any data, numerical or categorical, gives confidence values, and is easier to interpret. Then I started printing and reading papers on decision trees, random forests, and the bootstrap. I read about four papers a week. The random forest also improves on the standard decision tree, which I wrote about previously, on problems 1 and 7 above. 

I’ll post the code in a few days once it has been tested.