Data for Everyone? Self-service Data Integration

The notion of self-service data integration has been around for a while, somewhat hand-in-hand with self-service BI – both primarily coming from the perspective of power business users like data and business analysts. But a door also opened for less-technical LOB business users to participate in data integration activities. SaaS and cloud services introduced self-service data integration where SaaS providers or third-party vendors supply wizard-like UIs for basic pre-defined integrations or data synchs between specific applications. This version of self-service DI gained impetus from the Salesforce ecosystem of data integration services, many of which targeted “everyday” business users.

Data Integration as a Service seemed to be a new force for the democratization of data. However, when looking at current modern data integration solutions, it appears that tech-savvy business users are the main focus of self-service DI capabilities. So what has happened to data integration for everyday business users?

Data for Everyone?

The democratization of data has a place in Modern Data Integration, as both conceptual and actual future directions for solutions. Such democratization is meant in the context of sharing data more widely throughout the organization, connecting many more people to the data they need (not just upper-level executives). Organizations thrive and sustain success with better and faster availability of the right data at the right time.

The democratization of data includes self-service data integration and other data services, such as data preparation, that can give business users more hands-on access to data, frequently without the direct intervention of IT. But it would seem that there is quite a ways to go to actually connect different kinds of business roles to the appropriate self-service data integration tools. There has been way more talk than actual availability of such solutions.

Of course, it will take very sophisticated technology behind the scenes to keep it simple in the UIs that give everyday business users access to data integration capabilities. A lot of work must be done, especially to build in the safeguards to keep less technical business users out of trouble. To ensure that everyday business users do the work correctly and avoid serious missteps, self-service data integration has to be narrowly defined, with guidance at every step.

Cloud data integration and data prep solutions support dynamic data catalogs or libraries that provide “certified” data sets that can work well for self-service DI. The resulting data sets are available to the appropriate user roles, including the less technical. Governance of which roles access what data furthers the democratization of data in a risk-controlled environment. It’s important that business and IT teams continuously validate the contents of such data libraries, as many data sources come with a short shelf life of relevance.

Data Governance for Everyone

Self-service data integration is clearly moving forward, and it must go hand-in-hand with self-service governance. There is real risk associated with self-service DI that must be addressed through centralized IT management and governance, which need to be in place to prevent piecemeal and poorly managed efforts scattered throughout an organization. Cloud-based self-service data integration solutions are well-positioned for governance that maps into overall organizational data governance practices and policies.

But remember: data governance is more business than tech: business people are essential for implementing successful data governance. Once again, the crucial partnership between business and IT is a core piece of data governance to make sure that it will work well and make sense both for business needs and objectives, and IT management of self-service DI activities.

Data Prep Solutions – Blazing a Trail for Self-Service DI for ‘Everyone’?

Data preparation vendors have very effectively shared studies that highlight the costly nature of getting data ready for use by organizations: identifying the right sources; cleansing and validating data; testing for context and relevance; creating the appropriate integrations. Business and technology teams alike are better understanding why data integration and data preparation matter, and why they must be done well. These are not trivial undertakings and require sophisticated technologies to produce the usable data sets greatly needed for many business activities.

An extensive array of data preparation technologies have emerged to help connect more business people to the data they require for analytics and BI, seeking to greatly reduce onerous preparation time to generate that data. Data integration and data prep do have intertwined destinies since the goal of either should be reliable, business-ready data. Data prep draws on data quality processes that are have been very much part of data integration for quite some time, as has data lineage.

It looks like data prep solutions are creating the next level of tools that more quickly achieve reliable data, by utilizing applications of newer technologies like machine learning. Data prep offerings incorporate focused machine learning / algorithmic capabilities to automate more of data testing and quality as well as approaches to data integration.

Obviously, a core aspect of data preparation is self-service through SaaS/cloud platforms. The vendors behind these self-service offerings are clearly interested in extending the variety of roles, both business and technical, that can take advantage of data prep platforms. Right now the primary users are very technical DI experts and tech-savvy business analysts. But various vendors are working to include less-technical LOB business roles, as well as LOB developers. And these vendors do seem to understand that different user roles require different UIs, capabilities, guidance and governance, and access to data.

Self-Service Options and Modern Data Integration

We are seeing significant changes in integration processes, where data flows from and to anywhere: on-premises, cloud, partner systems, third-party enrichment, and so on. With Modern Data Integration, application and data integration are converging to provide everything organizations need in a comprehensive platform. Ultimately for business needs, it’s still all about the data.

And it’s about more people in the organization gaining faster access to data to help them do their jobs better, with improved effectiveness and innovation for the business. Through cloud-based platforms, self-service options can be available to all kinds of roles in an organization.

Self-service data integration is an important goal or Modern Data Integration. But self-service data integration – and preparation – still has a long road to travel before data can be accessed by “everyone”.