Agile Data Warehousing

I gave a talk on Agile Data Warehousing and met a bunch of great data folks at the recent DAMA Philadelphia chapter meeting. I always like to find out what is going on in other organizations, and it is refreshing to hear that people are starting to leverage Agile techniques to deliver data solutions. Agile has become a mainstay with our software engineering counterparts who have been using Scrum to quickly deliver enterprise software and mobile apps for years. For us data practitioners, we are still catching up and many have been misguided to think that an Agile methodology will not work in our environment.

The most common thing I hear is that you can’t build a warehouse using Agile because it is impossible to iterate over a data model. Modeling changes are going to happen regardless of the delivery methodology so learning to deal with change is important even if you are delivering in a waterfall environment. If change is going to happen, why not embrace it early and make it part of the way you work? As you start to deliver you should plan to:

Set expectations that you will build the base data structures first with rapid model changes occurring.
Wait to build complex presentation layers until the initial model has stabilized to reduce rework.
Coach the development team on how to write code that can withstand model volatility.
Manage expectations with leadership that things will break and code will be “thrown away”. Remind them that you are delivering business value, and if you can do that quickly some rework down the road will seem like a small price to pay.

Another challenge I often hear is that warehousing tasks can’t be broken up into small enough increments because of the complexity, unique data relationships and dependencies. Educating the team and the business about how data can be used can change this mindset. Having a beautiful dashboard in 3 months is great, but being able to get some data from a stage table in 3 weeks that will answer a key business problem is even better. Options such as temporary access to a staging environment are often never discussed and it could answer 80% of what was needed.

Challenge the team and the business to think differently about using data.No doubt that these are challenging problems, but they can be overcome and an Agile methodology can help. In my opinion, the biggest challenge to overcome is not technical, it is organizational. Getting the senior leadership team, the delivery team itself and the supporting IT organizations bought in to an Agile approach to delivering warehouse solutions requires a commitment to change that many companies have not been willing to make until now. The need for quicker access to information to make key business decisions is forcing organizations to innovate or risk being left behind. Becoming more Agile in building data solutions is not longer an option it is a necessity to stay competitive.