By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data science anayst
    Growing Demand for Data Science & Data Analyst Roles
    6 Min Read
    predictive analytics in dropshipping
    Predictive Analytics Helps New Dropshipping Businesses Thrive
    12 Min Read
    data-driven approach in healthcare
    The Importance of Data-Driven Approaches to Improving Healthcare in Rural Areas
    6 Min Read
    analytics for tax compliance
    Analytics Changes the Calculus of Business Tax Compliance
    8 Min Read
    big data analytics in gaming
    The Role of Big Data Analytics in Gaming
    10 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Delivering Data Warehousing and BI Projects using Agile
Share
Notification Show More
Latest News
ai in automotive industry
AI Is Changing the Automotive Industry Forever
Artificial Intelligence
SMEs Use AI-Driven Financial Software for Greater Efficiency
Artificial Intelligence
data security in big data age
6 Reasons to Boost Data Security Plan in the Age of Big Data
Big Data
data science anayst
Growing Demand for Data Science & Data Analyst Roles
Data Science
ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Mining > Delivering Data Warehousing and BI Projects using Agile
Business IntelligenceData MiningData Warehousing

Delivering Data Warehousing and BI Projects using Agile

bharden
Last updated: 2011/03/06 at 10:05 PM
bharden
8 Min Read
SHARE

There is a lot of buzz lately around Agile BI and Agile data warehousing.  Typical questions asked include – Does it produce better results?  Is it faster?  How much does it cost?  And of course, most importantly, is the Agile methodology a fit for Data Warehousing and Business Intelligence projects?

What is Agile?

There is a lot of buzz lately around Agile BI and Agile data warehousing.  Typical questions asked include – Does it produce better results?  Is it faster?  How much does it cost?  And of course, most importantly, is the Agile methodology a fit for Data Warehousing and Business Intelligence projects?

More Read

embedded bi tools

Embedded BI Tools Bring Huge Benefits to Business Applications

How IoT Can Be Connected to Business Intelligence
14 Brands Using Mobile Apps Instead of Ads to Build Customer Loyalty
5 Vital Business Intelligence Tips All Companies Should Embrace
Impressive Ways that AI Improves Business Analytics Insights

What is Agile?

Companies are marketing their BI and DW products as agile solutions, meaning that they are easy to work with, adaptable to business changes and quick to stand up (think BI data discovery tools and data warehouse appliances).  While the topic of an enterprise agile solution is worthy of a discussion there is another type of agile. In project management circles, when the word Agile comes up in conversation, it is always in the context of the Agile software development methodology, which also happens to be the context for this article. I won’t spend time discussing the Agile Manifesto or guiding principles here, instead I’ll try to talk about how Agile applies to BI/DW projects. If Agile techniques are new to you there are plenty of resources available on the web to learn more, withWikipedia being as good a starting point as any.

Why Choose Agile?

Does Agile produce better results than other methodologies when building a DW/BI application?  The only real answer is that it depends.  Building a BI application, a data warehouse or some other type of software is a complex process regardless of the methodology you choose.  It is a commonly quoted statistic that over 50% of projects fail and some studies have even measured failure rates as high as 80%. So does using Agile increase the success percentage?  The short answer is probably not. 

So if it does not improve project success rates then why bother?  If we are all familiar with traditional waterfall delivery methods, why not stick with what we know? There are two great answers to these question and which, in my opinion, confirm why we should all be delivering projects using Agile techniques.  First , Agile delivers business value to our customers quickly, which is critical as time to market is a competitive advantage.  For example, give the business access to customer data from the first source while you work on loading the second.  The business can start analyzing the available data immediately and as sources are added their results become richer.  Secondly, delivering functionality incrementally puts in a budgetary check and balance system.  If the project is going to be a failure you have an opportunity to make an assessment after each incremental delivery.  In a waterfall project there is a good chance you won’t know (or won’t admit) the project has failed until you are delivering it.  So while the total project cost may be similar at the end, there is a huge savings if you can recognize a project failure early.

Will it Work for Warehousing?

Agile may deliver value to customers faster, and may be more cost effective, but can it work for Business Intelligence and Data Warehouse projects?  Absolutely!  It is interesting that given the benefits it has not yet gained mainstream acceptance in the warehousing and reporting project space.  Agile has been slow to catch on because project teams are resistant to change, as more projects are successfully delivered I believe we will see Agile become the delivery methodology of choice.  Let’s start with why it makes sense in data warehousing.  A warehouse project plan can have iterations built in fairly easily if framed correctly.  Splitting delivery by subject areas and/or sources of data is a natural way to segment the schedule and provides an easy way to deliver value early and often.  The quicker we can get data loaded and into production, the faster data consumers can use data for decision making.  One potential downside to a subject area or source base approach is trying to implement without strong architectural and modeling guidelines in place.  Knowing how keys will be structured, how data quality issues will be handled and what the overall warehouse roadmap looks like is critical to an Agile delivery of a warehouse solution.  Upfront planning around modeling and architecture will smoothen subject or source based iterations.  As with all projects, communication is critical, and with an Agile delivery it is important to communicate the inevitability of rework as the warehouse evolves.  Setting these expectations ahead of time will pay dividends when the first major change is required.  Even with potential rework, the benefits gained by putting data in the hands of the business earlier more than pays for cost of future rework.

Will it Work for Business Intelligence?

Once a subject area or source data load is ready, the next step is to provide data to consumers using a Business Intelligence tool.  Once again, the Agile techniques are ideal for building and deploying a business intelligence application.  Building the semantic layer can be done in a similar fashion to developing the warehouse itself.  As long as a clear architecture roadmap has been set, the semantic layer can be a fast follower to the subject area or source file ETL load, perhaps even rolling out with the subject area itself.  Individual reports, dashboards and cubes can be managed and prioritized via a backlog and delivered as quickly as they are finished, adding incremental value on a regular basis.

Conclusions

Delivering a warehouse project using Agile will feel uncomfortable at first.  It probably won’t work perfectly and you may have to adapt the processes to work with your organizational polices and procedures.  Stick with it, listen to your team and don’t be afraid to make adjustments as you go (it is agile after all).  If you go in with an open mind and a clear roadmap Agile will deliver business value faster than you ever imagined possible.

TAGGED: agile, business intelligence, data warehousing
bharden March 6, 2011
Share this Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

ai in automotive industry
AI Is Changing the Automotive Industry Forever
Artificial Intelligence
SMEs Use AI-Driven Financial Software for Greater Efficiency
Artificial Intelligence
data security in big data age
6 Reasons to Boost Data Security Plan in the Age of Big Data
Big Data
data science anayst
Growing Demand for Data Science & Data Analyst Roles
Data Science

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

embedded bi tools
Business Intelligence

Embedded BI Tools Bring Huge Benefits to Business Applications

5 Min Read
internet of things and business intelligence
Internet of Things

How IoT Can Be Connected to Business Intelligence

6 Min Read
businesses using mobiles app for success
Big Data

14 Brands Using Mobile Apps Instead of Ads to Build Customer Loyalty

19 Min Read
tips for making the most of business intelligence
Business Intelligence

5 Vital Business Intelligence Tips All Companies Should Embrace

7 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence
AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?