By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data science anayst
    Growing Demand for Data Science & Data Analyst Roles
    6 Min Read
    predictive analytics in dropshipping
    Predictive Analytics Helps New Dropshipping Businesses Thrive
    12 Min Read
    data-driven approach in healthcare
    The Importance of Data-Driven Approaches to Improving Healthcare in Rural Areas
    6 Min Read
    analytics for tax compliance
    Analytics Changes the Calculus of Business Tax Compliance
    8 Min Read
    big data analytics in gaming
    The Role of Big Data Analytics in Gaming
    10 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Preprocessing – Feature Generation
Share
Notification Show More
Latest News
ai in automotive industry
AI Is Changing the Automotive Industry Forever
Artificial Intelligence
SMEs Use AI-Driven Financial Software for Greater Efficiency
Artificial Intelligence
data security in big data age
6 Reasons to Boost Data Security Plan in the Age of Big Data
Big Data
data science anayst
Growing Demand for Data Science & Data Analyst Roles
Data Science
ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Analytics > Predictive Analytics > Preprocessing – Feature Generation
Predictive Analytics

Preprocessing – Feature Generation

Editor SDC
Last updated: 2009/03/01 at 11:03 PM
Editor SDC
5 Min Read
SHARE

Price and volume data is the easiest to acquire and holds quite a bit of information. Preprocessing is the technique of making it easier to find patterns. The following are examples of different ways of processing price and volume series (with spreadsheet formulas in parentheses). Using raw data together with a few transformed series derived from the raw data as input features usually works best. Simply having a “universal” learner is not good enough in application.

Price – Dollars per share is of course the typical raw form
Alternatively price could be expressed as…

  • Percent above 52-week low, if you believe it is a support (p’ = p / min(p({365,0})) – 1)
  • Percent change from previous, if you believe it is autocorrelated (continuous: p’ = ln(p0/p1); arithmetic: p’ = (p0-p1)/p1)
  • Log price, if you believe it is a compounding process (p’ = ln(p))
  • Some other currency, if you want to remove some macroeconomic variable (p’ = p*CUR/USD)
  • Price points i.e. integer dollar value crosses like .99 -> 1.00 vs 1.23 -> 1.24, if you believe investors irrationaly weight large digit changes (p’ = floor(p0) ~= floor(p1))
  • Residuals vs an index, if you want to remove movement not unique to the security (p’ = …


Price and volume data is the easiest to acquire and holds quite a bit of information. Preprocessing is the technique of making it easier to find patterns. The following are examples of different ways of processing price and volume series (with spreadsheet formulas in parentheses). Using raw data together with a few transformed series derived from the raw data as input features usually works best. Simply having a “universal” learner is not good enough in application.

More Read

predictive analytics in dropshipping

Predictive Analytics Helps New Dropshipping Businesses Thrive

Promising Benefits of Predictive Analytics in Asset Management
Albanian Bitcoin Investors Tap the Power of Predictive Analytics
Predictive Analytics Improves Trading Decisions as Euro Rebounds
Can Predictive Analytics Help Traders Navigate Bitcoin’s Volatility?

Price – Dollars per share is of course the typical raw form
Alternatively price could be expressed as…

  • Percent above 52-week low, if you believe it is a support (p’ = p / min(p({365,0})) – 1)
  • Percent change from previous, if you believe it is autocorrelated (continuous: p’ = ln(p0/p1); arithmetic: p’ = (p0-p1)/p1)
  • Log price, if you believe it is a compounding process (p’ = ln(p))
  • Some other currency, if you want to remove some macroeconomic variable (p’ = p*CUR/USD)
  • Price points i.e. integer dollar value crosses like .99 -> 1.00 vs 1.23 -> 1.24, if you believe investors irrationaly weight large digit changes (p’ = floor(p0) ~= floor(p1))
  • Residuals vs an index, if you want to remove movement not unique to the security (p’ = p – DJIA)

Volume – shares per day/period is the typical raw format
Alternatively volume could be expressed as…

  • Volume minus the regressed volume of an index, if you want to compensate for the overall rise in volume though history (v’ = v – DJIA)
  • Percentage change just like price, above
  • Discretized as high, medium, low instead of a number, if you want to use classification rather than regression (v’ = bins(v, 3))
  • Volume minus average volume on current period, if you want to compare the morning with the afternoon or compare monday to tuesday without inserting another variable (v’ = v – periodavg(v))

Combinations of two series might also distil important information. Ex. price*volume, if you believe a price move is “confirmed” by high volume.

Preprocessing is a time consuming step because it requires domain-specific knowledge so a computer can’t do it efficiently and automatically. For ex. to find out that log price might be meaningful since companies grow organically a computer would have to test a huge library of basic functions: exp(p), p^k, sqrt(p), p*k, p+k, exp(-p), log(p) etc…

Technical indicators are just another way of preprocessing a time series. Often they take multiple points of data (such as a 30 day MA) and compress it to just one. This is useful for a human but less so for a computer which can computationally handle the full details. However it may be useful to add in common indicators to out-game human traders who are being influenced by these crude signals.

Editor SDC March 1, 2009
Share this Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

ai in automotive industry
AI Is Changing the Automotive Industry Forever
Artificial Intelligence
SMEs Use AI-Driven Financial Software for Greater Efficiency
Artificial Intelligence
data security in big data age
6 Reasons to Boost Data Security Plan in the Age of Big Data
Big Data
data science anayst
Growing Demand for Data Science & Data Analyst Roles
Data Science

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

predictive analytics in dropshipping
Predictive Analytics

Predictive Analytics Helps New Dropshipping Businesses Thrive

12 Min Read
analyst,women,looking,at,kpi,data,on,computer,screen
Predictive Analytics

Promising Benefits of Predictive Analytics in Asset Management

11 Min Read
predictive analytics helps Albanian bitcoin investors
Blockchain

Albanian Bitcoin Investors Tap the Power of Predictive Analytics

9 Min Read
benefits of data analytics for financial management
Predictive Analytics

Predictive Analytics Improves Trading Decisions as Euro Rebounds

10 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI and chatbots
Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Artificial Intelligence Chatbots Exclusive
ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?