Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    data analytics and truck accident claims
    How Data Analytics Reduces Truck Accidents and Speeds Up Claims
    7 Min Read
    predictive analytics for interior designers
    Interior Designers Boost Profits with Predictive Analytics
    8 Min Read
    image fx (67)
    Improving LinkedIn Ad Strategies with Data Analytics
    9 Min Read
    big data and remote work
    Data Helps Speech-Language Pathologists Deliver Better Results
    6 Min Read
    data driven insights
    How Data-Driven Insights Are Addressing Gaps in Patient Communication and Equity
    8 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Analysis of a Bad Indicator
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Mining > Analysis of a Bad Indicator
Business IntelligenceData Mining

Analysis of a Bad Indicator

Editor SDC
Editor SDC
5 Min Read
SHARE

I watched a video lecture, as I often do, on data analysis. here’s the video: the Hilbert Spectrum. Here are the notes I took while watching it:


The idea is appealing- to decompose a time series into underlying trends of different periodicities. In the trading world this would correspond to maybe a long term macroeconomic trend, a monthly pattern occurring around announcement of the federal funds rate, and a short term pattern caused by supply and demand and liquidity constraints. The researcher in the video was trying to study ocean waves with satellite data. Obviously there may be a difference in the two processes.

I implemented the Hilbert spectrum algorithm because I was excited about it. Here’s the R script. For example, here’s what the spectrum looks like for GOOG & TYP share prices:

At the top is the actual price series and below that are the series with the high frequency patterns removed one by one. They look nice.

Here’s the code, hspect.r, in the language R. R is basically an advanced calculator that’s also programmable.

More Read

Why IT Doesn’t “Get” Analytics (and Why the Time is Right for Change)
Guest post: Why Google TV Could Destroy Nielsen’s Data
Habits of Innovation
Another Step for Google into Business Analytics? EIM with Google Refine 2.0
Interactive Analysis and Relate Tools – Part I

The problem is that this is a type of smoother, useful for summarizing and exploring data, but useless for extrapolation or prediction. Among this …


I watched a video lecture, as I often do, on data analysis. here’s the video: the Hilbert Spectrum. Here are the notes I took while watching it:


The idea is appealing- to decompose a time series into underlying trends of different periodicities. In the trading world this would correspond to maybe a long term macroeconomic trend, a monthly pattern occurring around announcement of the federal funds rate, and a short term pattern caused by supply and demand and liquidity constraints. The researcher in the video was trying to study ocean waves with satellite data. Obviously there may be a difference in the two processes.

I implemented the Hilbert spectrum algorithm because I was excited about it. Here’s the R script. For example, here’s what the spectrum looks like for GOOG & TYP share prices:

At the top is the actual price series and below that are the series with the high frequency patterns removed one by one. They look nice.

Here’s the code, hspect.r, in the language R. R is basically an advanced calculator that’s also programmable.

The problem is that this is a type of smoother, useful for summarizing and exploring data, but useless for extrapolation or prediction. Among this family is cubic spline interpolation and LOESS. At the edges, if you extend these curves to make predictions the estimates will have extremely high variance. Making predictions with one of these smoothers is equivalent to throwing away almost all your data except the bit at the very end, and then either fitting a 3rd degree polynomial to it (in cubic spline interpolation) or a straight line (in LOESS).

Cubic spline interpolation is especially insidious because most people don’t understand it and a confusing name doesn’t help. Everyone knows how to interpret two derivatives: velocity and acceleration. The third derivative is interpretable, in two different contexts, as curvature or as burst. Burst is like if you’re standing in an elevator and it goes up, how much you feel it. If the elevator is designed will, burst
will be a constant and you will barely feel it. It’s also important in roller coaster design to ensure you have a smooth ride. In terms of curvature, if the third derivative is constant, it will be pleasing to the eye as if it were drawn by sweeping hand motions. That’s the qualitative explanation. This latter interpretation of curvature is what cubic spline interpolation is based on. The cubic spline
interpolation fits a nice-looking piecewise (between each two points) polynomial which matches 1st and 2nd derivatives at each knot.

Unfortunately you have to understand these methods to know not to use them and not to trust systems based on them. I’ve had people contact me about using cubic spline interpolation for prediction but it’s just not applicable.

Feel free to add your own thoughts.

TAGGED:data analysis
Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

AI Document Verification for Legal Firms: Importance & Top Tools
AI Document Verification for Legal Firms: Importance & Top Tools
Artificial Intelligence Exclusive
AI supply chain
AI Tools Are Strengthening Global Supply Chains
Artificial Intelligence Exclusive
data analytics and truck accident claims
How Data Analytics Reduces Truck Accidents and Speeds Up Claims
Analytics Big Data Exclusive
predictive analytics for interior designers
Interior Designers Boost Profits with Predictive Analytics
Analytics Exclusive Predictive Analytics

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

Spurring Growth in the UK Economy with Data Analysis

4 Min Read
data analysis
AnalyticsBig DataCRM

Customer Feedback and Data Analysis: The Keys to a Good Customer Retention Rate

6 Min Read

Hell is other people’s data

5 Min Read
automatic data analysis
Analytics

5 Huge Benefits of Automatic Data Analysis for SMEs

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

giveaway chatbots
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive
ai chatbot
The Art of Conversation: Enhancing Chatbots with Advanced AI Prompts
Chatbots

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?