Preprocessing – Feature Generation

Price and volume data is the easiest to acquire and holds quite a bit of information. Preprocessing is the technique of making it easier to find patterns. The following are examples of different ways of processing price and volume series (with spreadsheet formulas in parentheses). Using raw data together with a few transformed series derived from the raw data as input features usually works best. Simply having a “universal” learner is not good enough in application.

Price – Dollars per share is of course the typical raw form
Alternatively price could be expressed as…

• Percent above 52-week low, if you believe it is a support (p’ = p / min(p({365,0})) – 1)
• Percent change from previous, if you believe it is autocorrelated (continuous: p’ = ln(p0/p1); arithmetic: p’ = (p0-p1)/p1)
• Log price, if you believe it is a compounding process (p’ = ln(p))
• Some other currency, if you want to remove some macroeconomic variable (p’ = p*CUR/USD)
• Price points i.e. integer dollar value crosses like .99 -> 1.00 vs 1.23 -> 1.24, if you believe investors irrationaly weight large digit changes (p’ = floor(p0) ~= floor(p1))
• Residuals vs an index, if you want to remove movement not unique to the security (p’ = p – DJIA)

Volume – shares per day/period is the typical raw format
Alternatively volume could be expressed as…

• Volume minus the regressed volume of an index, if you want to compensate for the overall rise in volume though history (v’ = v – DJIA)
• Percentage change just like price, above
• Discretized as high, medium, low instead of a number, if you want to use classification rather than regression (v’ = bins(v, 3))
• Volume minus average volume on current period, if you want to compare the morning with the afternoon or compare monday to tuesday without inserting another variable (v’ = v – periodavg(v))

Combinations of two series might also distil important information. Ex. price*volume, if you believe a price move is “confirmed” by high volume.

More Read

Take the Predictive Analytics in the Cloud survey

Big Data, Unstructured Information Analysis is More Than Sentiment.

5 Ways Big Data Is Transforming Finance

Amazon: Using Big Data Analytics to Read Your Mind

Big Analytics: Closing the ‘Clue Gap’ with Big Data

Preprocessing is a time consuming step because it requires domain-specific knowledge so a computer can’t do it efficiently and automatically. For ex. to find out that log price might be meaningful since companies grow organically a computer would have to test a huge library of basic functions: exp(p), p^k, sqrt(p), p*k, p+k, exp(-p), log(p) etc…

Technical indicators are just another way of preprocessing a time series. Often they take multiple points of data (such as a 30 day MA) and compress it to just one. This is useful for a human but less so for a computer which can computationally handle the full details. However it may be useful to add in common indicators to out-game human traders who are being influenced by these crude signals.