Voodoo Spectrum of Machine Learning and Data Sets
468 Views
I used to be very gung-ho about machine learning approaches to trading but I’m less so now. You have to understand that that there is a spectrum of alpha sources, from very specific structured arbitrage opportunities -> to stat arb -> to just voodoo nonsense.
As history goes on, hedge funds and other large players are absorbing the alpha from left to right. Having squeezed the pure arbs (ADR vs underlying, ETF vs components, mergers, currency triangles, etc) they then became hungry again and moved to stat arb (momentum, correlated pairs, regression analysis, news sentiment, etc). But now even the big stat arb strategies are running dry so people go further, chasing mirages (nonlinear regression, causality inference in large data sets, etc).
In modeling the market, it’s best to start with as much structure as possible before moving on to more amorphous statistical strategies. If you have to use statistical machine learning, encode as much trading domain knowledge as possible with specific distance/neighborhood metrics, linearity, variable importance weightings, hierarchy, low-dimensional factors, etc.
It’s good to have a heuristic feel for the …
I used to be very gung-ho about machine learning approaches to trading but I’m less so now. You have to understand that that there is a spectrum of alpha sources, from very specific structured arbitrage opportunities -> to stat arb -> to just voodoo nonsense.
As history goes on, hedge funds and other large players are absorbing the alpha from left to right. Having squeezed the pure arbs (ADR vs underlying, ETF vs components, mergers, currency triangles, etc) they then became hungry again and moved to stat arb (momentum, correlated pairs, regression analysis, news sentiment, etc). But now even the big stat arb strategies are running dry so people go further, chasing mirages (nonlinear regression, causality inference in large data sets, etc).
In modeling the market, it’s best to start with as much structure as possible before moving on to more amorphous statistical strategies. If you have to use statistical machine learning, encode as much trading domain knowledge as possible with specific distance/neighborhood metrics, linearity, variable importance weightings, hierarchy, low-dimensional factors, etc.
It’s good to have a heuristic feel for the danger/flexibility/noise sensitivity (synonyms) of each statistical learning tool. I roughly have this spectrum in my head:
Very specific, structured, safe
Optimize 1 parameter, require crossvalidation↓Optimize 2 parameters, require crossvalidation↓Optimize parameters with too little data, require regularization↓Extrapolation↓Nonlinear (SVM, tree bagging, etc)↓Higher-order variable dependencies↓Variable selection↓Structure learning
Very general, dangerous in noise, voodoo
This diagram is worth expanding. If anyone has any suggestions, please leave them.
Trending Now
1
Big Data
51 shares2,885 views
51 shares2,885 views
Fascinating Ways Big Data Is Reshaping The Future of Football
Sean Mallon - September 8, 20182
Big Data
31 shares2,622 views
31 shares2,622 views
How Big Data And Education Can Work Together To Help Students Thrive
Ryan Ayers - September 7, 20183
Blockchain
77 shares2,666 views
77 shares2,666 views
Bringing Together Blockchain, eCommerce and Content
Daniel Matthews - September 6, 20184
Big Data
36 shares2,476 views
36 shares2,476 views
The Telecom Industry Needs Big Data To Thrive In The 21st Century
Annie Qureshi - September 6, 20185
Big Data
1,981 views
1,981 views
Maximizing your Influencer Marketing Campaign with Big Data
Rehan Ijaz - September 6, 20186
Machine Learning
38 shares2,379 views
38 shares2,379 views
An Introduction To Machine Learning Using Spark Language
Nirmal Patel - September 5, 20187
Big Data
81 shares2,831 views
81 shares2,831 views
You must log in to post a comment.