By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data analytics in sports industry
    Here’s How Data Analytics In Sports Is Changing The Game
    6 Min Read
    data analytics on nursing career
    Advances in Data Analytics Are Rapidly Transforming Nursing
    8 Min Read
    data analytics reveals the benefits of MBA
    Data Analytics Technology Proves Benefits of an MBA
    9 Min Read
    data-driven image seo
    Data Analytics Helps Marketers Substantially Boost Image SEO
    8 Min Read
    construction analytics
    5 Benefits of Analytics to Manage Commercial Construction
    5 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Exterior Design, Part II: Frequency
Share
Notification Show More
Latest News
data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security
ai in software development
3 AI-Based Strategies to Develop Software in Uncertain Times
Software
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Quality > Exterior Design, Part II: Frequency
Data QualityData WarehousingExclusive

Exterior Design, Part II: Frequency

Rob Armstrong
Last updated: 2011/04/03 at 12:34 PM
Rob Armstrong
7 Min Read
SHARE

To continue the theme of designing for an ever changing set of requirements, I wanted to talk about the second item we all know will happen.  Exactly how and when it will happen is open for debate but it will happen, unless you prevent it.  The second known is that the users will want to have the data more frequently.  For now I am referring to the same data as opposed to increasing the granularity of the data (that comes a bit later).  The best way to illustrate this is with a true story from my travels.

To continue the theme of designing for an ever changing set of requirements, I wanted to talk about the second item we all know will happen.  Exactly how and when it will happen is open for debate but it will happen, unless you prevent it.  The second known is that the users will want to have the data more frequently.  For now I am referring to the same data as opposed to increasing the granularity of the data (that comes a bit later).  The best way to illustrate this is with a true story from my travels.

I was in South America visiting a retail account.  I asked about the frequency of the data they have to analyze.  The business lead in the meeting responded that they get data every month.  The IT lead responded they get data every day.  This is quite a disparity.  After a bit of back and forth, the IT lead sighed heavily and said “we load daily data every month”.  I had to shake my head as I pointed out that if that was the case, they only had monthly data.  Once again we have to take the user perspective and getting May 5th data in June does not help you make decisions on May 7th.  So what was the problem in getting data that supported the end user needs?

More Read

data analytics in sports industry

Here’s How Data Analytics In Sports Is Changing The Game

Advances in Data Analytics Are Rapidly Transforming Nursing
Data Analytics Technology Proves Benefits of an MBA
Anti-Spoofing is Crucial for Data-Driven Businesses
3 AI-Based Strategies to Develop Software in Uncertain Times

Editor’s note: Rob Armstrong is an employee of Teradata. Teradata is a sponsor of The Smart Data Collective.

In this case it was a matter of the historically processes put in place by the IT organization.  In order to load the data the tables need to be offline.  Since the total monthly load took over 48 hours (including the summary table and index maintenance as well) to be accomplished they could not run this job frequently.  Much of this was a legacy problem but no one ever bothered to either optimize the processes using newer techniques or calculate that running the job with a much smaller daily amount would not take the long downtime.  Again it was simply a case of this is how we do it and it is too hard to change.

So we can reasonably expect that the users will ask for their data to become fresher until it reaches the point that critical business data will be required with “real time”.  I actually do not like term as it means something completely different for many different parts of an organization.  I like to say that if the frequency of data loading has to be the half life of a user’s ability to respond to it.  Therefore, if a user can get a piece of data and take a direct action on it within 2 minutes then they should be getting the data updated every minute.  This is not a hard and fast rule but it at least establishes a starting point for the conversations.

Keeping our “exterior” perspective it also must be noted that the loading is not the end point.  The data has to be readily available to the end user so any ancillary maintenance to take it from loaded into detail to being accessible to queries must also be included.

This gets to another one of those interesting paradoxes.  Things like indexing, summary tables, cubes, and what not are suppose to improve performance for the end users.  Unfortunately as the data frequency trends to “real time” those very same “performance optimizations” become a barrier and will actually increase the latency from data available to data accessible.

So as in last installment, I’ll ask a question.  If you are in IT, how ready are you for the community to ask that the data become “more frequent”?  If you load weekly, are you ready for daily?  If you load daily are you ready for hourly?  

If you are in the business arena, what could you do to increase the value of your processes if you got the data “more frequently”?  Better inventory management, better response to customer problems, decrease the lag enough that you can implement analytics into the call center?  Finally, if you are in the executive or leadership roles, what would these improved processes mean in terms of increase revenue or decreased costs?

OK, one more readily known requirement to be designed for and that will be the next blog topic.

Finally, thank you to all those who provided feedback and comments on my original efforts.  I trust that over the next few postings I’ll find my voice and give you all a place to pick up some useful ideas and insights.

Rob Armstrong April 3, 2011
Share this Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

data analytics in sports industry
Big Data

Here’s How Data Analytics In Sports Is Changing The Game

6 Min Read
data analytics on nursing career
Analytics

Advances in Data Analytics Are Rapidly Transforming Nursing

8 Min Read
data analytics reveals the benefits of MBA
Analytics

Data Analytics Technology Proves Benefits of an MBA

9 Min Read
anti-spoofing tips
Security

Anti-Spoofing is Crucial for Data-Driven Businesses

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI and chatbots
Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Artificial Intelligence Chatbots Exclusive
giveaway chatbots
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?