We use cookies, including third-party cookies from Google to serve personalized ads through AdSense, to operate this site and understand how it is used. By continuing to browse, you accept this use. See our Privacy Policy and Terms of Use for details, including how to opt out of personalized advertising.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
    Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
    23 Min Read
    Flat editorial illustration: The article is about automating the collection, preparation, and delivery of marketing data across p
    From Data to Dashboards: 5 Marketing Reporting Automation Software Picks
    12 Min Read
    Flat editorial illustration: The article contrasts passive follower totals with a behavioral path from attention and trust to cli
    Creator Analytics Beyond Follower Count: Seven Metrics That Measure Audience Behavior
    11 Min Read
    Flat editorial illustration: The article connects AI and marketing analytics with managerial judgment grounded in consumer behavi
    How an Online MBA in Marketing Connects AI and Data Analytics
    11 Min Read
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers -- AI-generated illustration
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers
    7 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Automate the Boring But Essential Parts of Your Data Warehouse
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Mining > Automate the Boring But Essential Parts of Your Data Warehouse
AnalyticsBusiness IntelligenceData ManagementData MiningData QualityData VisualizationData WarehousingDecision ManagementPredictive AnalyticsRisk ManagementSoftwareUnstructured Data

Automate the Boring But Essential Parts of Your Data Warehouse

Keith Peterson
Keith Peterson
5 Min Read
Automate the Boring But Essential Parts of Your Data Warehouse
Illustration generated with FLUX.2 [klein 4B] via Cloudflare Workers AI.
SHARE

If you are in IT and responsible for your company’s data warehouse and reporting capabilities, chances are you will identify with at least one of these statements:

If you are in IT and responsible for your company’s data warehouse and reporting capabilities, chances are you will identify with at least one of these statements:

More Read

CIOs Predict IT Development
CIOs Predict IT Development
Game Analytics: Platform Trends, Monetization and Player Value in Social Games
Benefits of Using Analytics to Optimize Your Telemarketing Strategy  
Marketing and IT Get Together: The Elephants in the Room
Interview: Dr Graham Williams
  • “We are too buried with maintaining our existing database to take on new initiatives”
  • “We need to get out of the report writing business”
  • “We cannot give you that information from the data warehouse without a lot of development”
  • “It’s too hard to make changes to our existing process because it breaks too easily”
  • “That new data source doesn’t fit in our existing data model and we can’t extend it” 

If so, then you may be an ideal candidate for Data Warehouse Automation (DWA). Companies realize that data warehouses are not going to be replaced by Hadoop or in-memory solutions.  A 2014 Gartner survey found that only 3% of IT leaders believe that big data or in-memory systems can replace their existing data warehouse infrastructure. That’s a dramatic drop over recent years.  As such, DWA is getting a fresh look as a better way to build and manage a data warehouse.

Until recently, DWA was associated mostly with automating ETL development – such as generating SSIS packages in the Microsoft environment.  Today, however, it covers all the major components of data warehousing from design, development and testing to deployment, operations and change management.  It also covers advanced functionality like support for slowly changing dimensions and change data capture.

In our experience, DWA delivers up to 80% improvements in the cost-effectiveness of building and running a data warehouse.  And, just as important, DWA is far better aligned with modern agile development practices because it encourages a rapid, iterative approach to design.

The benefits of automation are profound for IT organizations.   Developers and DevOps teams see greater efficiency in the development and maintenance phases of the data warehouse lifecycle through a number of advantages: 

Data Warehouse Development

Data Warehouse Operations

 

  • Greater productivity for developers means more delivered in less time
  • Consistency of practices and standards leads to the development of more maintainable systems
  • Automation means increased support for agile development practices
  • Standardized testing processes supports QA cycles
  • Simpler development and prototyping makes it easier to respond to changing requirements
 
  • Maintenance processes are simplified and packaged to reduce manual effort
  • Documentation is generated automatically and kept current with releases
  • Assess impact of proposed DW changes using enhanced metadata capabilities
  • Add enterprise features to basic data warehouses to increase DW lifespan
  • More robust, standard processes leads to a more stable environment

DWA is commonly criticized by vendor competitors as being either 1) “the wrong approach” compared to in-memory; or 2) not as effective as hand-crafted, custom database development approaches.  Both arguments are purist and too simplistic. In-memory has tremendous analytical benefits but is lacking in support for data governance, single source of truth, and performance.  Customization is almost always needed for business critical tasks – but not for the more common routine and repetitive uses.  Custom development is more error prone than DWA, and much harder to maintain over the long run. 

Automation does not mean excluding in-memory analytical tools or precluding customization.  It means blending the best of the best to create a scalable, high performing solution at lower time and development cost.

Next Generation Data Warehouse

To deliver on your company’s future demands for data and insights, you will need to maintain your existing data warehouse – and add the great new capabilities available with big data management and in-memory analytics. The real opportunity is in making those technologies work together smoothly with minimum effort and risk.  That means continuing to automate the rote and routine parts of your developer’s daily task so they can focus on moving your organization to a higher level of development and value, and ultimately provide better access to the data that is valuable to the company.

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

Flat editorial illustration: The article connects downstream report failures, data-quality gaps, and conflicting metric definitio
Signs Your Data Architecture Needs an Outside Review
Big Data Exclusive
Data-Driven Pultrusion: Best GFRP Rebar Machines & ROI
Data-Driven Pultrusion: Best GFRP Rebar Machines & ROI
Big Data Exclusive
Flat editorial illustration: The article contrasts tools for gathering fresh interviews with tools for analyzing existing qualita
10 AI Market Research Tools That Turn Qualitative Data into Insights
Artificial Intelligence Big Data Exclusive
Flat editorial illustration: The article compares proxy MCP servers by what an AI agent can do through them: manage proxy account
Proxy MCP Servers: 7 Options and What Each Lets an AI Agent Do
Artificial Intelligence Big Data Data Collection

Stay Connected

1.2KFollowersLike
33.7KFollowersFollow
222FollowersPin

You Might also Like

Visier Workforce Analytics After the Show
AnalyticsBest Practices

Visier Workforce Analytics After the Show

5 Min Read
Think Big – Applying Analytics to Injury Claims Is the Next Challenge for Law Firms
Big Data

Think Big – Applying Analytics to Injury Claims Is the Next Challenge for Law Firms

8 Min Read
Do Social Media Monitoring Tools Provide True Intelligence?
AnalyticsBusiness IntelligenceData MiningDecision ManagementMarket ResearchMarketing

Do Social Media Monitoring Tools Provide True Intelligence?

5 Min Read
data access for engineers
Big Data

How The Explosive Growth Of Data Access Affects Your Engineer’s Team Efficiency

11 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence
5 Great Tips for Using Data Analytics for Website UX
5 Great Tips for Using Data Analytics for Website UX
Big Data

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-26 SmartData Collective. All Rights Reserved.
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?